cmd/go: 'go test' fails to vet runtime/cgo when gcc not installed

[georgysavva]

What version of Go are you using (go version)?

$ go version
go version go1.15.5 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build679279315=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am running go test during the docker image build. My codebase consists of a single Go package and doesn't have any cgo code. Here is my Dockerfile:

FROM golang:1.15.5 as build

WORKDIR /go/src/app

# Cache Go dependencies
COPY go.mod go.sum ./
RUN go mod download

COPY . .

# This is needed to prove that the Go standard library gets recompiled. 
# Since in order to recompile it requires gcc. 
RUN apt-get remove -y gcc
RUN go test ./...

What did you expect to see?

I expected to see that go test successfully finishes after running my tests

What did you see instead?

go test failed with this message:

# runtime/cgo
exec: "gcc": executable file not found in $PATH
FAIL    github.com/georgysavva/driver-app/gateway/pkg/gateway [build failed]
FAIL    github.com/georgysavva/driver-app/gateway/pkg/gateway2 [build failed]

I've noticed that if I add go build -i -a std before the go test ./... command. It no longer needs to recompile:

FROM golang:1.15.5 as build

WORKDIR /go/src/app

# Cache Go dependencies
COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN go build -i -a std
# This is needed to prove that the Go standard library doesn't get recompiled.
RUN apt-get remove -y gcc
RUN go test ./...

go test ./... finishes successfully in that case.

My question is what is the condition for go test ./... to recompile that standard library, because if I remove any of those go build flags: -i or -a the solution is no longer works. And why would go test ./... need to rebuild the standard library on a fresh Go installation at all?
BTW, the situation is the same if I do go vet ./... instead of go test ./....

I encountered this issue the first time with golang alpine image(golang:1.15.5-alpine3.12) because it doesn't have gcc installed. So this is pretty related to #28065. Maybe it's worth adding that command go build -i -a std to the official golang images to fix that kind of problem in user images that inherit them?
Thanks!

[jayconrod]

I wasn't able to reproduce this using the Dockerfile you gave. Could you please post:

• go env output before running a command that rebuilds the standard library (if different from the go env output already posted)
• go list -f '{{if .Stale}}{{.ImportPath}} {{.StaleReason}}{{end}}' std at the same point in time.

In general, a standard library package needs to be recompiled if something in the global configuration that changes, for example, if you set CC or CGO_ENABLED=0 or -tags, that triggers a rebuild. The go list -json output should say whether a package is stale and why.

[georgysavva]

Hi and thanks for your message!
Here is the additional information that you asked for:

go env Output before running go test ./...

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/go/src/app/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build578085697=/tmp/go-build -gno-record-gcc-switches"

Output of go list -f '{{if .Stale}}{{.ImportPath}} {{.StaleReason}}{{end}}' std was empty.

Here is also the link to the codebase that contains the Dockerfile that I posted earlier, so you reproduce it with my code:
https://github.com/georgysavva/driver-app/tree/go-std-recompile/gateway

In general, a standard library package needs to be recompiled if something in the global configuration that changes, for example, if you set CC or CGO_ENABLED=0 or -tags, that triggers a rebuild. The go list -json output should say whether a package is stale and why.

Yeah I understand that. But I use a fresh official Golang image and don't change anything in the global configuration so there should be nothing to trigger a rebuild.

[jayconrod]

Thanks for posting that repo. It doesn't look like the standard library is actually being recompiled with go test ./..., but an action is failing on runtime/cgo because gcc is missing.

It looks like it's actually vet that's failing. I see no errors when building with go test -c -vet=off. There are also no issues if I pre-populate the build cache with go vet std before removing gcc.

go test automatically runs a subset of the go vet tests for the packages named on the command line and their dependencies. vet analyzes source files passed to the compiler, which includes cgo generated sources. A C compiler is needed to run cgo. Vet results and cgo generated sources are stored in the build cache, but they aren't part of the Go installation, so that's why you might see failures when running go test without -vet=off.

I think we should probably close this, but cc @bcmills and @matloob in case they have other thoughts.

[bcmills]

If running go vet on a user's pure-Go package requires cgo source files for other imported packages, then we should either fix vet not to do that or find a way to ship the C source files matching the compiled export data along with our installation. I don't think it's reasonable to require users to install gcc just to generate source code for vet to analyze for packages that are already built and aren't part of their module.

[georgysavva]

@jayconrod yeah, you are right, vet is causing the problem and if you do go vet std before go test ./... the problem is gone. But as I mentioned earlier running go build -i -a std does the same (gcc is no longer required for go test ./...) and I wonder why go build -i -a std helps in this situation.

@bcmills I totally agree with you. And similar thoughts have already occurred to other users: #28065 (comment), #28065 (comment)

[jayconrod]

vet needs to type-check files given the compiler, and that needs to happen on cgo-generated go files. I don't think we should include those files in the distribution, but it would be reasonable to skip running the standard vet passes enabled in go test for pre-compiled packages in std if a C compiler isn't available.

@georgysavva go build -a forces packages to be recompiled, even if they're already installed and up-to-date. As part of that recompilation, the cgo-generated go files that vet is missing will be written to the build cache. They'll be available later, even after gcc is uninstalled.

[bcmills]

@jayconrod

it would be reasonable to skip running the standard vet passes enabled in go test for pre-compiled packages in std if a C compiler isn't available

I agree with that, but I think it's only tangentially related.

The OP listed go test ./... as the command to reproduce the failure. That should not be matching any package in std, so it shouldn't result in a vet run against any package in std.

I suspect that vet is re-typechecking dependencies imported by the named packages from source instead of reading their type information from export data. It probably should not be doing that.

[georgysavva]

@jayconrod thanks for clarifying this!

Thanks for looking into this, I hope you will find the reason.