cmd/go: tests fail when GOROOT is stale

[mvdan]

I was running go test inside cmd/go, just like one does, and was surprised to see a failure:

$ go test -run=TestScript/mod_outside
go test proxy running at GOPROXY=http://127.0.0.1:39569/mod
--- FAIL: TestScript (0.00s)
    --- FAIL: TestScript/mod_outside (0.96s)
        script_test.go:257: 
            # This script tests commands in module mode outside of any module.
            #
            # First, ensure that we really are in module mode, and that we really don't have
            # a go.mod file. (0.003s)
            # 'go list' without arguments implicitly operates on the current directory,
            # which is not in a module. (0.006s)
            # 'go list' in the working directory should fail even if there is a a 'package
            # main' present: without a main module, we do not know its package path. (0.004s)
            # 'go list all' lists the transitive import graph of the main module,
            # which is empty if there is no main module. (0.003s)
            # 'go list' on standard-library packages should work, since they do not depend
            # on the contents of any module. (0.166s)
            # 'go list' should work with file arguments. (0.032s)
            # 'go list' on a package from a module should fail. (0.003s)
            # 'go list -m' with an explicit version should resolve that version. (0.006s)
            # 'go list -m -versions' should succeed even without an explicit version. (0.005s)
            # 'go list -m all' should fail. "all" is not meaningful outside of a module. (0.003s)
            # 'go list -m <mods> all' should also fail. (0.003s)
            # 'go list -m <mods>' should fail if any of the mods lacks an explicit version. (0.003s)
            # 'go list -m' with wildcards should fail. Wildcards match modules in the
            # build list, so they aren't meaningful outside a module. (0.006s)
            # 'go clean' should skip the current directory if it isn't in a module. (0.003s)
            # 'go mod graph' should fail, since there's no module graph. (0.003s)
            # 'go mod why' should fail, since there is no main module to depend on anything. (0.002s)
            # 'go mod edit', 'go mod tidy', and 'go mod fmt' should fail:
            # there is no go.mod file to edit. (0.008s)
            # 'go mod download' without arguments should report an error. (0.003s)
            # 'go mod download' should download exactly the requested module without dependencies. (0.007s)
            # 'go mod download all' should fail. "all" is not meaningful outside of a module. (0.003s)
            # 'go mod vendor' should fail: it starts by clearing the existing vendor
            # directory, and we don't know where that is. (0.003s)
            # 'go mod verify' should fail: we have no modules to verify. (0.003s)
            # 'go get' has no go.mod file to update outside a module and should fail. (0.014s)
            # 'go get' should not download anything. (0.006s)
            # 'go build' without arguments implicitly operates on the current directory, and should fail. (0.003s)
            # 'go build' of a non-module directory should fail too. (0.003s)
            # 'go build' of source files should fail if they import anything outside std. (0.036s)
            # 'go build' of source files should succeed if they do not import anything outside std. (0.053s)
            # 'go build' should succeed for standard-library packages. (0.042s)
            # 'go build' should use the latest version of the Go language. (0.040s)
            # 'go doc' without arguments implicitly operates on the current directory, and should fail.
            # TODO(golang.org/issue/32027): currently, it succeeds. (0.006s)
            # 'go doc' of a non-module directory should also succeed. (0.006s)
            # 'go doc' should succeed for standard-library packages. (0.010s)
            # 'go doc' should fail for a package path outside a module. (0.023s)
            # 'go install' with a version should succeed if all constraints are met.
            # See mod_install_pkg_version. (0.175s)
            # 'go install' should fail if a package argument must be resolved to a module. (0.004s)
            # 'go install' should fail if a source file imports a package that must be
            # resolved to a module. (0.031s)
            # 'go install' should succeed with a package in GOROOT. (0.219s)
            > go install cmd/addr2line
            [stderr]
            go install cmd/addr2line: cmd/go/internal/work/exec.go:1648: testgo must not write to GOROOT (installing to GOROOT/pkg/tool/linux_amd64/addr2line)
            [exit status 1]
            FAIL: testdata/script/mod_outside.txt:207: unexpected command failure
FAIL

There isn't anything special about my GOROOT; it's just built from source.

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/mvdan/.cache/go-build"
GOENV="/home/mvdan/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/mvdan/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/mvdan/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/mvdan/tip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/mvdan/tip/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="devel go1.18-6656269288 Thu Sep 30 01:32:54 2021 +0000"
GCCGO="gccgo"
GOAMD64="v3"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/mvdan/tip/src/cmd/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1046648373=/tmp/go-build -gno-record-gcc-switches"

[bcmills]

This looks to me like a symptom of a stale GOROOT, and it isn't currently showing up on the builders.

Does it go away if you re-run make.bash? (Do you perhaps have GOROOT_FINAL set?)

[bcmills]

Aha, I think this may have to do with GOAMD64 being set.

[mvdan]

You're right, on a clean make.bash it succeeds. My GOROOT wasn't clean because I checked out a branch containing a change to go fmt, so I guess that modified the go binary and by extension the toolchain's action ID and such.

Surely the test should be resilient enough against a stale GOROOT? Typically I haven't felt the need to do a go install std cmd loop until the build cache is happy before continuing to develop - the cache just figures it out as needed. A full make.bash at every git checkout also seems unfortunate.

I'm not sure if GOAMD64 is related. Maybe? I can't reproduce with the same environment (including GOAMD64) and a clean build :)

[bcmills]

The cmd/go tests definitely requite a clean GOROOT. Otherwise, it would be possible to run make.bash, then go test std cmd, and end up with a GOROOT that is substantially different from what you would get if you just stopped at make.bash. (Compare #30316, #28387.)

We probably ought to have ! stale checks in a bunch more places than we do, but we can't easily add them today in part because of #33598. 😩

[mvdan]

That's fair. I still think that reasonably running into a failure is still surprising, so I assume this issue is somewhat valid even if not urgent :)

[bcmills]

Rearchitecting the tests to avoid writing to GOROOT even when it is stale would be an awful lot of work, and would probably make the cmd/go tests a lot slower. It's probably better to have them fail explicitly than to have them succeed but take way longer than they should...