New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: per-test setup and teardown support #27927

Closed
dfawley opened this Issue Sep 28, 2018 · 13 comments

Comments

Projects
None yet
5 participants
@dfawley

dfawley commented Sep 28, 2018

There are some checks that are helpful to perform after every test runs, e.g. monitoring for leaked goroutines. However, remembering to add these checks to every single test in a package is difficult and cumbersome. The testing package should provide a way to allow setup and cleanup code to be run globally for every test in a package, which has access to the testing.T for each test case. Note that TestMain does not allow you to run code between test functions; only at the very beginning and end of the whole package.

@gopherbot gopherbot added this to the Proposal milestone Sep 28, 2018

@gopherbot gopherbot added the Proposal label Sep 28, 2018

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Sep 28, 2018

As a proposal, it would be helpful to suggest exactly how this should work, so we can see what new API would be required.

@dfawley

This comment has been minimized.

dfawley commented Sep 28, 2018

There are many different ways this could work, and I don't want to prescribe one for fear of flailing wildly and having it be rejected when other ways of implementing the same feature would work. A few ideas:

  1. An interceptor-style function with the signature func(t *testing.T, f func(*testing.T)) could be registered in an init(). The interceptor is given the t and the test function f, and can perform setup, call f, and then perform cleanup.

  2. A specially-named function (e.g. TestInterceptor(*testing.I) where I is a new interceptor type that contains t and f, as above.

  3. New functionality in testing.M used via TestMain.

The last one sounds the most approachable to me, so I will flesh it out a bit.

How I would imagine the usage would look:

package my_test

func TestMain(m *testing.M) {
	for _, t := range m.Tests {
		setup()
		t.Run()  // invokes the current test function
		if err := cleanup(); err != nil {
			t.Error("error cleaning up test:", err)
		}
	}
}

Implementation:

package testing

type M struct {
	...
	Tests []TestRunner
}

type TestRunner struct {
	*T  // embed the testing.T that will be used for this test.
	// other unexported fields as needed
}

func (TestRunner) Run()  // runs this test with this TestRunner's *T

I haven't looked into the implementation of the testing package much; it's possible an iterator instead of a slice would be easier to implement, since creating all the testing.Ts at initialization might be difficult/impossible -- or an initializer on the TestRunner to create the T could be required instead. Also, a TestRunner.Done method may be required for implementation detail reasons. (Again, an iterator might be better, since it could automatically perform any necessary cleanup after the previous test.)

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 3, 2018

/cc @mpvl
This seems too intrusive to me, fwiw.

@rsc

This comment has been minimized.

Contributor

rsc commented Dec 12, 2018

Various of us who have had to write code like this have found it helpful to write it explicitly instead of having an implicit thing that applies to every test in the package. Sometimes only 9 of 10 tests need a particular setup/teardown. Another common pattern is to put setup/teardown in one Test and then put the tests that need that setup/teardown into subtests. Or write a helper that does the setup/teardown and pass in the 'body' to run in between. There are lots of ways. We shouldn't hard-code one.

@rsc rsc closed this Dec 12, 2018

@dfawley

This comment has been minimized.

dfawley commented Dec 12, 2018

Per-test setup and teardown is a common feature in most testing frameworks, and is supported in Java, C++, and Python, in OSS and at Google. I know there are other ways to implement this, but they are all cumbersome or error-prone or both. Something included in the language's primary testing framework would make it both easy to use and reliable.

@jadekler

This comment has been minimized.

Contributor

jadekler commented Dec 12, 2018

+1. This is something extremely common in test frameworks. @rsc, please reconsider. This might not be something y'all use, but it is something the community uses:

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Dec 12, 2018

Why not just use a single test function with subtests? Why is that any harder to use than the suggestions in this issue?

@dfawley

This comment has been minimized.

dfawley commented Dec 12, 2018

@ianlancetaylor What would that look like for a package with 10~100 tests? Something like this is what I was imagining:

func testFoo(t *testing.T) { ... }
func testBar(t *testing.T) { ... }
func testBaz(t *testing.T) { ... }

func TestEverything(t *testing.T) {
	// Maintaining this map is error-prone and cumbersome (note the subtle bug):
	fs := map[string]func(*testing.T){"testFoo": testFoo, "testBar": testBar, "testBaz": testBar}
	// You may be able to use the `importer` package to enumerate tests instead,
	// but that starts getting complicated.
	for name, f := range fs {
		setup()
		t.Run(name, f)
		teardown()
	}
}

Is there a better way to do this?

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Dec 13, 2018

var tests = []func(t *testing.T){
    func(t *testing.T) {
        ...
    },
    func(t *testing.T) {
        ...
    },
    ...
}

func TestEverything(t *testing.T) {
    for i, fn := range tests {
        setup()
        t.Run(strconv.Itoa(i), fn)
        teardown()
    }
}

Yes, it's more awkward, but it seems manageable, and there are cases where a bit of awkwardness is preferable to hidden magic.

@dfawley

This comment has been minimized.

dfawley commented Dec 13, 2018

There are several undesirable aspects to your variant:

  • The tests are not named. That makes it difficult to individually run them, and when one fails, you don't really know what failed. With any more than 2-3 tests, it's hard to even find the test case you're looking for since the numbers are implicit. There's also the issue of re-numbering due to inserting/deleting tests.
  • The tests all must be in the same _test.go file. This approach doesn't work for larger packages with tests in multiple files.
  • The tests themselves are not top-level functions. That adds an extra level of indent everywhere. This is bad for complex tests.
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Dec 13, 2018

There are obvious mitigations for those comments, but I said up front that the approach is awkward.

On the other hand, implicit test setup and teardown adds actions that are not clearly visible when looking at a large testsuite. And a single package will often have different kinds of tests, some of which need a particular kind of setup and teardown and some of which need a different kind, so over time the complexity of the setup and teardown tend to increase.

I don't think there is an obvious win here so we are choosing to be explicit and simple, as is the common pattern for the Go language.

@dfawley

This comment has been minimized.

dfawley commented Dec 13, 2018

The 3rd approach proposed in #27927 (comment) is explicit, simple, and no more magical than any test that uses the existing TestMain functionality. Essentially, it proposes the ability to easily enumerate the test functions in the package, and run them individually instead of as a batch.

Just because some test suites don't want per-test setup/cleanup for all tests in the package doesn't mean that all test suites don't. I'd like to ensure that all tests in grpc check for goroutine leaks (similar to net/http's checks). We have almost 800 end-to-end + grpc package tests. As of now, only ~200 have this leak check. With per-test setup/cleanup functionality, I could guarantee 100% compliance, and improve my test coverage and ability to isolate and find bugs. If we do it manually, we need to visit hundreds of test functions to add it, and also hope that everyone remembers to include the check going forward. The suggestion to use subtests is extremely unwieldy at this kind of scale as well, and would require us to introduce our own magic that makes our tests no longer explicit and simple.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Dec 13, 2018

I suggest that you try writing a new proposal that changes the testing.M structure to provide a list of tests with some way to run them. That is, change the focus from a functionality that has some clear difficulties to a smaller, simpler change that lets you do what you want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment