cmd/go: using both -v and -json with go test has unintended effects

[firelizzard18]

Go version

go version go1.23.1 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/firelizzard/.cache/go-build'
GOENV='/home/firelizzard/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/firelizzard/go/pkg/mod'
GOOS='linux'
GOPATH='/home/firelizzard/go'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/firelizzard/sdk/go1.23.1'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/firelizzard/sdk/go1.23.1/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.1'
GODEBUG=''
GOTELEMETRY='on'
GOTELEMETRYDIR='/home/firelizzard/.config/go/telemetry'
GCCGO='/usr/bin/gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/firelizzard/src/yaegi/go-script/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4165056435=/tmp/go-build -gno-record-gcc-switches'

What did you do?

go test -v -json -run=- -bench=^BenchmarkSimple$ ./pkg/vm in the root of https://gitlab.com/firelizzard/go-script

What did you see happen?

Nothing but output events for the benchmark

What did you expect to see?

An "Action":"run" event for the benchmark

[gabyhelp]

Related Issues

• cmd/go: 'go test -json -bench' misattributes benchmark output to prior tests #27764 (closed)
• cmd/test2json: go test -bench -json -count output is inconsistent #66825
• go test ./... -v No output #69220 (closed)
• cmd/test2json: benchmark output lines are attributed to the preceding test #49505 (closed)
• testing: odd interaction between -json flag and TestMain #31969 (closed)
• stdout messes up `go test --json` #36576 (closed)
• Benchmark results vastly different when Tests are skipped with -run=XXX #42696 (closed)
• cmd/go: test -json issues #22734 (closed)
• cmd/test2json: missing pass/fail action on individual benchmark #61767
• cmd/go: -json test report ordering #28953

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[firelizzard18]

This is probably related to #61767 and CL 443596. It appears that the presence of -v results in the binary not being called with -test.v=test2json, preventing CL 443596's fix from working.

[dmitshur]

This is documented at https://pkg.go.dev/cmd/test2json as:

The test should be invoked with -test.v=test2json. Using only -test.v (or -test.v=true) is permissible but produces lower fidelity results.

Note that "go test -json" takes care of invoking test2json correctly [...]

In other words, just go test -json does the right thing (it does go test -json -v=test2json under the hood), and go test -json -v (aka go test -json -v=true) is permitted but produces the lower fidelity results you're observing. I'm not sure why it's permitted, though, leaving it to cmd/go owners to see if anything should/can be done.

CC @matloob, @samthanawalla.

[firelizzard18]

I guess instead of calling this a bug another way of putting it is: As a developer working on tooling (improved test support for vscode-go), it is unpleasantly surprising that adding -v degrades the utility of -json. I'd never use -v -json manually but it's not hard to miss when I'm shoving flags into an array in preparation for executing go test.