proposal: testing: add support for individual test function execution - expose testing.M.Tests

[mattdbush]

Expose collection of defined Tests for individual function test execution so that boiler plate code can be added around execution of test.

Example:

func TestMain(m *testing.M) {
	setupAll()
	for t, _ := range m.Tests {
		setup(t)
		result := t.Run()
		if result.Passed {
			// handle passed test, discard resources etc
		}
		if result.Failed {
			// example scenario - forward log for immediate notification/action
			// preserve/copy resources for troubleshooting inspection.
		}
		teardown(t)
	}
	teardownAll()
	os.Exit(code)
}

[seankhliao]

Duplicate of #27927

[mattdbush]

@seankhliao the original issue #27927 was only closed due to "age". It was and is still useful, please note that many other frameworks and languages have the feature requested on #53701, which is to enable code to be written before and after execution of each individual test executes. Why do you/go team feel this feature is not useful in the go context?

[ianlancetaylor]

Reopening.

That said, this proposal needs to clarify the type of the elements of m.Test. The sample code is calling t.Run, but that is clearly not the method testing.T.Run.

[rsc]

We've talked in the past - #28592 (comment) - about exposing a list of tests and letting TestMain reorder and filter it. I'm not as sure about exposing .Run itself. What if the test calls t.Parallel? It doesn't seem like TestMain should be getting in the way of actual test execution.

Note that you can get this kind of per-test setup and teardown already by making a single test - func Test for example - and then give it subtests that you call t.Run for, checking the result. Parallel still throws a wrench into all of this.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

If this issue is about per-test setup and teardown it seems like that is already provided by wrapping t.Run.
In that case it seems like there's nothing more to do here.

If that's not right, and there's more to do here, what exactly should we do? What API should we add and why?

[meling]

Looking at the initial example, there is an idea for inspecting the result of individual tests. I’ve long wanted a way to do that for my autograder tool. Currently, I have to carefully insert code before every t.Error and t.Fatal. Having a way to inspect the result of a test programmatically would allow doing my thing only once… instead of for every test failure location.

Not sure if that’s what OP wanted with the proposal.

[rsc]

@meling, If you call the tests with t.Run (as subtests), t.Run returns a boolean saying whether the test failed or not.

[rsc]

It sounds like maybe there's nothing to do here.

[mattdbush]

@rsc the use of t.Run is used in context of running sub-tests that are specific to a topic or functional area. The test startup and shutdown code wanted by this proposal is to provide common/cross-cutting concerns, and would want these functions applied to all unit tests and potentially all sub-tests.

So for example if we wanted to forward the test log for all individual tests (or all failed tests) to a communication service such as Slack and do this immediately when each test is executed. At present we have to parse log at end and forward at end of test runs. So rather than interact with unit test output as a whole logged file we could interact with test results and log output individually each test.

The benefit sought by this proposal is to avoid adding lots of boiler plate code inside our tests and around our sub-tests but to define common behaviours at a top level which improves the testing effectiveness and would make our CI/CD smoother.