cmd/go: using both -v and -json with go test has unintended effects

[firelizzard18]

Go version

go version go1.23.1 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/firelizzard/.cache/go-build'
GOENV='/home/firelizzard/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/firelizzard/go/pkg/mod'
GOOS='linux'
GOPATH='/home/firelizzard/go'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/firelizzard/sdk/go1.23.1'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/firelizzard/sdk/go1.23.1/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.1'
GODEBUG=''
GOTELEMETRY='on'
GOTELEMETRYDIR='/home/firelizzard/.config/go/telemetry'
GCCGO='/usr/bin/gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/firelizzard/src/yaegi/go-script/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4165056435=/tmp/go-build -gno-record-gcc-switches'

What did you do?

go test -v -json -run=- -bench=^BenchmarkSimple$ ./pkg/vm in the root of https://gitlab.com/firelizzard/go-script

What did you see happen?

Nothing but output events for the benchmark

What did you expect to see?

An "Action":"run" event for the benchmark

[gabyhelp]

Related Issues

• cmd/go: 'go test -json -bench' misattributes benchmark output to prior tests #27764 (closed)
• cmd/test2json: go test -bench -json -count output is inconsistent #66825
• go test ./... -v No output #69220 (closed)
• cmd/test2json: benchmark output lines are attributed to the preceding test #49505 (closed)
• testing: odd interaction between -json flag and TestMain #31969 (closed)
• stdout messes up `go test --json` #36576 (closed)
• Benchmark results vastly different when Tests are skipped with -run=XXX #42696 (closed)
• cmd/go: test -json issues #22734 (closed)
• cmd/test2json: missing pass/fail action on individual benchmark #61767
• cmd/go: -json test report ordering #28953

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[firelizzard18]

This is probably related to #61767 and CL 443596. It appears that the presence of -v results in the binary not being called with -test.v=test2json, preventing CL 443596's fix from working.

[dmitshur]

This is documented at https://pkg.go.dev/cmd/test2json as:

The test should be invoked with -test.v=test2json. Using only -test.v (or -test.v=true) is permissible but produces lower fidelity results.

Note that "go test -json" takes care of invoking test2json correctly [...]

In other words, just go test -json does the right thing (it does go test -json -v=test2json under the hood), and go test -json -v (aka go test -json -v=true) is permitted but produces the lower fidelity results you're observing. I'm not sure why it's permitted, though, leaving it to cmd/go owners to see if anything should/can be done.

CC @matloob, @samthanawalla.

[firelizzard18]

I guess instead of calling this a bug another way of putting it is: As a developer working on tooling (improved test support for vscode-go), it is unpleasantly surprising that adding -v degrades the utility of -json. I'd never use -v -json manually but it's not hard to miss when I'm shoving flags into an array in preparation for executing go test.

[matloob]

I'm not sure what we should do here. We could disallow passing -v and -json together to prevent users from accidentally getting the lower fidelity results, but I wonder if that would potentially unnecessarily break workflows?

[firelizzard18]

Would it be problematic to simply ignore -v if -json is set? Currently -v -json passes -v to the test binary. I am proposing that instead -v -json passes -v=test2json to the test binary. In either case, go test emits JSON events. The only real difference in behavior is: with -v=test2json the test binary emits framing markers that improve test2json's ability to parse test events. The current behavior is broken IMO. Once CL 601535 is merged, nested tests (like Go's script tests) will be escaped such that test2json will stop choking on them. With the current behavior of -v -json, that escaping is disabled and thus test2json will again choke on nested tests. If -v is simply ignored when -json is present, the behavior goes back to being sane.