Add support for "Coverage profiling support for integration tests"

[sfc-gh-kleonhard]

In 1.20, Go introduced support for integration test profiling: https://go.dev/testing/coverage/#gomodselect. Can support for this be added to rules_go so that coverage is collected from Go binaries executed during bazel coverage?

My use case: I've got go_test rules that depend on go_binary artifacts via the data attribute. The tests run the binaries using exec.Command. I'd like to aggregate test coverage from my code included in the binary under test.

Can I tell bazel coverage to set the -cover flag on any go_binary rules invoked and have it aggregate coverage?

Here's an example I'd like to work with: bazel coverage --combined_report=lcov //my_test

# running coverage tests should include coverage
# from the invocation of my_binary_a from my_test
go_library(
    name = "my_binary_under_test_lib",
    srcs = ["my_binary_under_test.go"],
    importpath = "github.com/example/my_binary_under_test",
)
go_binary(
    name = "my_binary_under_test",
    embed = [":my_binary_under_test_lib"],
)

go_test(
    name = "my_test",
    srcs = [
       # Runs my_binary via exec.Command
        "my_test.go",
    ],
    # These binaries are executed by my_test.go
    data = [
        "//my_binary_under_test",
    ],
)

I'd expect the resulting coverage to include coverage for my_binary_under_test.go as it's used in my_binary_under_test which is invoked by my_test.

[fmeum]

This is definitely something I would like to see, but it requires some work: Since we interface with Go on a lower level than go build, we need to figure out how coverage instrumentation is enabled on that level. We also need to call our cover to lcov conversion routine in the right places.

If anyone wants to work on this, I can offer my help. I am also in touch with the Go team and hope that they can provide some pointers.

[fmeum]

Leaving some pointers here for anybody who is interested in picking this up.

Exit hooks can be realized with https://github.com/golang/go/blob/8edcdddb23c6d3f786b465c43b49e8d9a0015082/src/runtime/coverage/hooks.go#L42.
Coverage instrumentation is handled by https://github.com/golang/go/blob/8edcdddb23c6d3f786b465c43b49e8d9a0015082/src/cmd/go/internal/work/exec.go#L2024.

[sfc-gh-kleonhard]

Thanks for the quick response, Fabian! I hope some ambitious gopher takes you up on your offer.

[fmeum]

Adding info I received from @thanm:

You mentioned that coverage collection has its own kind of exit code. Could you point me to how that is done?

In the new implementation, when you do "go build -cover -o myprogram.exe " to produce an instrumented binary, as part of compiling the "main" package, the compiler injects a call to runtime/coverage.initHook:

https://go.googlesource.com/go/+/f46320849da89bea3e23bae985ad753d30bbc5da/src/runtime/coverage/hooks.go#28

this call does two things: adds an exit hook that will run when the program terminates, and writes out a meta-data file (since the meta-data doesn't vary from run to run, we can write it out early).

2. rules_go doesn't use "go build", it directly runs "go tool compile". Could you explain to me how coverage instrumentation is controlled on that level?

This is a more complex question, but here are the basics. [Note that as a way of seeing more detail, I recommend doing a "go build -cover" of a toy program and passing "-x -work" to "go build", then capturing the output. You can then look in more detail a just what the go command is doing].

Let's say you have a program with a single package "main". When you build the program with "-cover", what happens under the hood is as follows:

For each package to be instrumented, the Go command first runs the cover tool, passing in a couple of config files with parameters it needs. Example (this is from the "go build -x -work" output):

 cat >/tmp/go-build253156559/b001/pkgcfg.txt << 'EOF'
 {"OutConfig":"/tmp/go-build253156559/b001/coveragecfg","PkgPath":"cov.example/main","PkgName":"main","Granularity":"perblock","ModulePath":"cov.example","Local":false}
 EOF
 cat >/tmp/go-build253156559/b001/coveroutfiles.txt << 'EOF'
 /tmp/go-build253156559/b001/main.cover.go
 EOF
 cd $WORK/b001/
 $GOROOT/pkg/tool/linux_amd64/cover \
    -pkgcfg ./pkgcfg.txt \

-mode set
-var goCover_c959797d4aa5_
-outfilelist ./coveroutfiles.txt
main.go

The contents of the file passed to "-pkgcfg" is a JSON-encoded instance of this struct: https://go.googlesource.com/go/+/f46320849da89bea3e23bae985ad753d30bbc5da/src/internal/coverage/cmddefs.go#12. It has things like the package name and package import path, the containing Go module, etc.

Especially to note is the value of the field "OutConfig". This is that path of the file that the cover tool is going to write, which the Go command is going to then pass on to the compiler.

Worth noting that in the new model, the cover tool is processing all files in the package at a single go (as opposed to instrumenting them one at a time).

Here's the compile command that builds the instrumented source produced by the cover tool:

$GOROOT/pkg/tool/linux_amd64/compile -o $WORK/b001/pkg.a -trimpath "$WORK/b001=>" -p main -lang=go1.18 -complete -buildid PPxaCSsbM6hx-gMi95WR/PPxaCSsbM6hx-gMi95WR -goversion go1.20.2 -coveragecfg=$WORK/b001/coveragecfg -c=4 -nolocalimports -importcfg $WORK/b001/importcfg -pack $WORK/b001/main.cover.go

Note the "-coveragecfg" option. This is simply taking the file produced in the previous cover step and feeding it directly into the compiler.

rules_go needs to inject code that converts coverage information in the native Go format to LCOV.

The way things work at the moment is that an instrumented binary just writes out a bunch of files, then the assumption is that the user will pick up those files and run tools on them to produce reports.

In your case you could write a stand-alone tool that would consume the files and use the results to produce LCOV data, or we could figure out some way to bypass the code that writes the files, and you could have code that processes the data on the fly at the point where the program is exiting.

For the latter strategy, we could use something like this prototype CL:

http://go.dev/cl/c/go/+/443975

This CL provides hooks for converting the new cov data at runtime back into the same form that the Go 1.19 cover tool uses, so if you have code that can digest the legacy cover vars, that might work for you. If so, we would need to figure out how to get the CL productionized and submitted.

[ceejatec]

I'm very interested in this functionality as well. I'd like to add that for our purposes, we would like the ability to perform normal compilation steps with the equivalent of "go build -cover -coverpkg /...." available - we don't want this integrated with existing "bazel coverage" runs, because the whole point of this new feature is to allow running integration tests while gathering coverage information. Integration testing would not normally be driven by bazel. Hopefully that is something that will come out of this effort as well. Thanks!

[sluongng]

It seems like golang/go#55953 was pushed to 1.23 so we can still use the old coverage for 1.22.

But we should start floating some design ideas for this so that we could have something 6 months from now.