cmd/go/internal/test: compile error for go test on package with no test files does not output package-level "FAIL" error

[joyzheng]

What version of Go are you using (go version)?

$ go version
go version go1.20.6 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jzheng/Library/Caches/go-build"
GOENV="/Users/jzheng/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/jzheng/Plaid/go/pkg/mod"
GONOPROXY="github.plaid.com"
GONOSUMDB="github.plaid.com"
GOOS="darwin"
GOPATH="/Users/jzheng/Plaid/go"
GOPRIVATE="github.plaid.com"
GOPROXY="https://proxy.golang.org"
GOROOT="/usr/local/Cellar/go/1.20.6/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.20.6/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.20.6"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="0"
GOMOD="/Users/jzheng/Plaid/go/src/github.plaid.com/plaid/go.git/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mw/swflvd316dl67xycd2lmfzhcll7jtg/T/go-build3323119429=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Run go test ./pkg ./pkg2 on the following directory:

go.mod
pkg/main.go
pkg2/main.go

go.mod:

module foo

go 1.21.4

pkg/main.go:

package pkg

import "fmt"

func main() {
	fmt.Println("Hello, world")
}

package pkg2/main.go (does not compile)

package pkg2

import "fmt"

func main() {
	fmt.Println("Hello, world")
	foo
}

What did you expect to see?

# foo/pkg2 [foo/pkg2.test]
pkg2/main.go:7:2: undefined: foo
?   	foo/pkg	[no test files]
FAIL	foo/pkg2 [build failed]
FAIL

The above is the output that shows up if I add a pkg2/main_test.go file with a line `package pkg2 line in the file.

What did you see instead?

go test ./pkg ./pkg2
# foo/pkg2
pkg2/main.go:7:2: undefined: foo
?   	foo/pkg	[no test files]
FAIL

The FAIL foo/pkg2 [build failed] line is missing. This is true in both regular and -json output formats, which makes it harder to parse the results of CI runs on many packages to see which packages resulted in failures.

[mknyszek]

CC @bcmills @matloob per https://dev.golang.org/owners

[ksw2000]

This problem does not seem to occur in the latest development version of Go.

go 1.20

> go version
go version go1.20.1 windows/amd64
> go test ./pkg ./pkg2
# foo/pkg2
pkg2\main.go:7:2: undefined: foo
note: module requires Go 1.22
?       foo/pkg [no test files]
FAIL

go 1.22 dev

> ../bin/go version
go version devel go1.22-5f7a408563 Wed Nov 22 02:20:04 2023 +0000 windows/amd64
> ../bin/go test ./pkg ./pkg2
# foo/pkg2
pkg2\main.go:7:2: undefined: foo
?       foo/pkg [no test files]
FAIL    foo/pkg2 [build failed]
FAIL

I thought that the older version printed the [build failed] message into a buffer and printed the [no test files] message to stdout.

go/src/cmd/go/internal/test/test.go

Lines 1236 to 1246 in 789b3f8

	if a.Failed {
	// We were unable to build the binary.
	a.Failed = false
	a.TestOutput = new(bytes.Buffer)
	fmt.Fprintf(a.TestOutput, "FAIL\t%s [build failed]\n", a.Package.ImportPath)
	base.SetExitStatus(1)
	// release next test to start
	close(r.next)
	return nil
	}

go/src/cmd/go/internal/test/test.go

Lines 1262 to 1265 in 789b3f8

	if p := a.Package; len(p.TestGoFiles)+len(p.XTestGoFiles) == 0 {
	fmt.Fprintf(stdout, "? \t%s\t[no test files]\n", p.ImportPath)
	return nil
	}

However, the new version prints the [build failed] message to stdout instead of to a buffer.

go/src/cmd/go/internal/test/test.go

Lines 1250 to 1258 in 5b123ae

	if a.Failed {
	// We were unable to build the binary.
	a.Failed = false
	fmt.Fprintf(stdout, "FAIL\t%s [build failed]\n", a.Package.ImportPath)
	// Tell the JSON converter that this was a failure, not a passing run.
	err = errors.New("build failed")
	base.SetExitStatus(1)
	return nil
	}