proposal: cmd/go: breakout testmain generation functionality into its own tool

[nemith]

When go test is ran there is logic in go/src/cmd/go/internal/load/test.go that will scan a package for it's given tests and generate the code for the entry point a test binary. The go tool then compiles and executes that binary.

External build tools like Bazel (https://bazel.build/) and Buck (https://buckbuild.com/) bypass the go tool and execute the compiler and linkers directly to get control over the output archives, etc to use for distributed caching and to unify building across multiple languages. To deal with tests both Buck[1] and Bazel[2] have done a copy and paste of the testmain generation code with some modifications and embedded them.

Being an internal package, obviously it is not a stable API. Every Go version there can be minor changes to how tests are discovered or how the testmain code generation is code. These have to be painstakingly backported into Bazel and Buck to keep working while also maintaining comparability with older Go versions.

Since the generated testmain code is linked with the testing library found in the stdlib it would make sense that there is a clear way to generate testmain code that is tied to the version of Go toolchain.

I am proposing abstracting out the test discovery and testmain generation code into it's own external tool (e.g go tool testmaingen) that can be called from external tools. The go tool could call either a shared library or the testmaingen tool directly.

This would be similar to how cgo code generation is done.

[1] https://github.com/facebook/buck/blob/master/src/com/facebook/buck/features/go/testmaingen.go.in
[2] https://github.com/bazelbuild/rules_go/blob/master/go/tools/builders/generate_test_main.go

[nemith]

cc: @bcmills @rsc @ianlancetaylor

[bcmills]

CC @jayconrod @matloob @ianthehat

[josharian]

Somewhat related: #29430

[jayconrod]

You can kind of already do this. If you run go list -test -f '{{if eq .Name "main"}}{{.GoFiles}}{{end}}' *.go, go list writes the testmain source to the build cache, and you can read it out from there. Kludgy though.

Something that complicates this change is that we generate the testmain source when loading packages, not during execution as you'd expect. The generated testmain is listed in .GoFiles, not .CompiledGoFiles because of this. If we were going to generate the testmain in a subprocess, we should move the code generation into a proper action first so that we can benefit from caching. Actually, we should do this anyway so go list can do less work.

Even if we did this, I'd be very surprised if the generated file could be used as-is. Bazel needs to do some other stuff in main before running tests, like changing directories, sharding, filtering, and coverage post-processing. Every build system is going to need some custom code, and I'm not confident we can factor all that out into a separate file.

[nemith]

That is pretty interesting. I didn't know it was stored in the cache. Digging through the cache seems like it could be more fragile than what we are dealing with here.

I'd be very surprised if the generated file could be used as-is. Bazel needs to do some other stuff in main before running tests, like changing directories, sharding, filtering, and coverage **post-processing.

In Buck we change directories , do test discovery, do sharding through a "test runner" which is just a wrapper around the compiled testmain which seems like a much cleaner solution with better abstractions, but that isn't really the topic here.

Does Bazel/Blaze do this for other languages as well like Python or C++?

Every build system is going to need some custom code, and I'm not confident we can factor all that out into a separate file.

There is very little that Buck does here intentionally as it's already too hard to maintain the testmain generation code with just changes to the testing package

The core problem we have had w/ Buck is that the testing package and the generated code that uses has changed with nearly every major Go release. We have been able to hack around this by adding reflection, implementing an interface, etc Even if some of the testmaingen code was available as an external library that could be imported it would help, but it would require a stable API between it and the testing package which may be less than ideal.

Here are some of the changes that has been required.

• facebook/buck@4271ef7#diff-d7a1345bc79619677ed4da43c65e7540
• facebook/buck@2b454ad#diff-d7a1345bc79619677ed4da43c65e7540

In fact when we upgrade to new Go versions this is really the only consistent blocker we have had over the years.

[rsc]

I think @jayconrod's answer here is the right one. There shouldn't be a separate API for generating testmain.go by itself. Even knowing that testmain.go is a thing is too much detailed knowledge about tests. What we've exposed is how to ask 'tell me the exact way the test binary should get built' instead. That's a much more general query.

[andybons]

Per discussion with @golang/proposal-review, looks like this is resolved. Please comment if that’s not the case.

[nemith]

I guess then the problem is there is no spec on how other build tools should interact with the testing package. It would be beneficial to have a spec on how that interaction would look so that other building tools could just build their own test main generation (or whatever).

The original problem is that the interaction between the the testing package and the tool is very much intertwined leading to a lot of breakage when you step slightly outside of it.