proposal: testing: output timing and outcome of TestMain or possible setup and teardown functions

[doron-cohen]

Hi,

We are currently using the TestMain function to setup and teardown test-containers in our integration tests. This allows us to prepare the environment once for the entire test suite and save time.
We also monitor our test CI step by using gotestsum which practically parses the output from go test -json. This lets us keep the CI snappy.

One issue we are having is that there is no way to monitor TestMain's timing or outcome. In gotestsum there is some code to add a fake test case in the case the failure occurred in TestMain (see comment by @dnephin ). But, there is still no way to monitor how long did the logic before and after the tests ran.

There are several possible ways to tackle this IMO:

Declaring setup and teardown functions

Introduce new functions for setup and teardown:

func TestMain(m *testing.M) {
	m.Setup(setupFunc)
	m.Teardown(teardownFunc)
}

This can work just like testing.T.Cleanup does. I think this is beneficial for other use cases where there is a need to better organize testing code.

Emitted JSONs could look like:

{"Time":"2022-12-10T14:13:37.4857284-02:00","Action":"pass","Package":"example.com/cool/app","Function":"setupFunc","Elapsed":0.0321}

Monitor TestMain timing without m.Run time

Another option is to time the entire TestMain but stop the timer on m.Run and resume when it ends.

Hypothetical code:

func (m *testing.M) Run() int {
	m.timer.Stop()
	defer m.timer.Continue()

	# ... run tests
}

Emitted JSONs could look like:

{"Time":"2022-12-10T14:13:37.4857284-02:00","Action":"pass","Package":"example.com/cool/app","Function":"TestMain","Elapsed":10.3853}

[seankhliao]

note testing.M.Cleanup was previously declined in #42161

[doron-cohen]

note testing.M.Cleanup was previously declined in #42161

Thanks for the reference. I think there is another valid use case here which I didn't see in that proposal. Also, we can achieve TestMain timing without these as I suggested.

[kegsay]

note testing.M.Cleanup was previously declined in #42161

I believe this was erroneously declined, on the basis that:

It seems to me that testing.M.Cleanup can be implemented simply using defer.

This is not true. Consider:

package test

import (
	"fmt"
	"os"
	"testing"
)

func TestMain(m *testing.M) {
	defer func() {
		fmt.Println("cleanup code")
	}()
	os.Exit(m.Run())
}

func TestFoo(t *testing.T) {
	t.Fatalf("oh no")
}

Cleanup code does not execute because of os.Exit(), which is called because the docs tell you to:

Run runs the tests. It returns an exit code to pass to os.Exit.

Let's ignore that for now then. Consider this other code:

package test

import (
	"fmt"
	"testing"
)

func TestMain(m *testing.M) {
	defer func() {
		fmt.Println("cleanup code")
	}()
	m.Run()
}

func TestFoo(t *testing.T) {
	panic("oh no")
}

In this scenario, the defer does not run at all.

It's simply not true that you can get cleanup functions working at the testing.M level with the current tooling.

[adabuleanu]

It's simply not true that you can get cleanup functions working at the testing.M level with the current tooling.

@kegsay argument is true. I experienced the same behavior. Any progress on this feature or any alternatives on working around this scenario?