Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: 'go test' on alpine and ubuntu requires gcc in GOPATH mode #28065

Open
mvdan opened this issue Oct 7, 2018 · 28 comments

Comments

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 7, 2018

What version of Go are you using (go version)?

golang:1.11.1-alpine3.8:

go version go1.11.1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

Also from that docker image:

GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build610617865=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Run go test on a simple package that imports some network packages like net/http. Expect it to work, even when I don't have gcc installed, which is the case with the Alpine Go images. This worked on 1.10. It broke when we switched to the 1.11 images.

Here is a simple reproducer script:

docker run -i golang:1.11.1-alpine3.8 sh <<-SCRIPT
        mkdir -p src/foo.com/bar
        cd src/foo.com/bar

        cat <<EOF >main.go
        package main

        import _ "net/http"

        func main() {}
        EOF

        go test
SCRIPT

What did you expect to see?

What I see if I use golang:1.11.1-stretch:

$ ./repro.sh
?       foo.com/bar     [no test files]

Note that this has nothing to do with the package not having test files or tests. We encountered this on a package that has plenty of tests.

What did you see instead?

$ ./repro.sh
# runtime/cgo
exec: "gcc": executable file not found in $PATH

Also note that we are not in module-aware mode, so this issue should be different from #26988. I believe this is the same issue that @mfridman and @tylercloke were reporting in that thread.

For the time being, we've just globally set CGO_ENABLED=0 on all of our CI jobs, since we don't use CGO on our production builds anyway. And one could argue that we should have been doing that from the start anyway. Nonetheless, this still seems like a bug or some sort of regression to me.

/cc @bcmills @thepudds @myitcv

@mvdan mvdan added this to the Go1.12 milestone Oct 7, 2018
@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Oct 7, 2018

Forgot to mention - the stretch image does have gcc, while the alpine image does not.

@adamdecaf

This comment has been minimized.

Copy link
Contributor

@adamdecaf adamdecaf commented Oct 7, 2018

This might be better classified as "the official golang:alpine images should install a supported libc" .

From #27264 (comment)

The official Go binaries are known to not work on Alpine because we assume glibc. That is #18773 and #19938.

Are you saying that the Go binary from go get golang.org/dl/go1.10.3 but not from go get golang.org/dl/go1.11? Or did you get your go1.10.3 from somewhere else? (e.g. from Alpine apk)

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Oct 7, 2018

I don't think this is related. I'm not having any issues with Alpine's musl libc, and Go does otherwise work.

If the official Go docker image based on Alpine required installing glibc, what would be the point of distributing that image to begin with?

@FiloSottile

This comment has been minimized.

Copy link
Member

@FiloSottile FiloSottile commented Oct 9, 2018

Also, I don't think we make any statement as to requiring glibc for Go to work if properly built. That comment is about the Go binary distribution.

@palsivertsen

This comment has been minimized.

Copy link

@palsivertsen palsivertsen commented Oct 10, 2018

Looks like a dupe of #27639

@mtibben

This comment has been minimized.

Copy link

@mtibben mtibben commented Oct 12, 2018

I think this issue should stay open to describe the go test issue as opposed to the go build issue. See #26988 (comment).

@mtibben

This comment has been minimized.

Copy link

@mtibben mtibben commented Oct 12, 2018

I'm also seeing this behaviour when running go test on Windows (no gcc)

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Oct 12, 2018

There have been a bunch of related issues, some being fixed and some being closed as duplicates. Note however, that many of them including #26988 require running the Go command in module-aware mode, while this one doesn't.

It does look like #27639 is a very similar issue, but that was closed and didn't have much activity, so I think we should keep this one open for now.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Oct 24, 2018

There's something about the Alpine image that makes the go command think rebuilding is necessary. Does pkg/linux_amd64/runtime/cgo.a even exist in the image?

If the Alpine image were built with CGO_ENABLED=0 then I could see that doing a CGO_ENABLED=1 build would do a rebuild. But I don't see that in the Dockerfile.
https://github.com/docker-library/golang/blob/ed78459fac108dab72556146b759516cc65ee109/1.11/alpine3.8/Dockerfile

@palsivertsen

This comment has been minimized.

Copy link

@palsivertsen palsivertsen commented Oct 25, 2018

pkg/linux_amd64/runtime/cgo.a exists yes.

Dump of runtime and http files in the docker image:

# tree pkg/linux_amd64/runtime* pkg/linux_amd64/net/http* pkg/linux_amd64/vendor/golang_org/x/net/http*
pkg/linux_amd64/runtime
├── cgo.a
├── debug.a
├── internal
│   ├── atomic.a
│   └── sys.a
├── pprof
│   └── internal
│       └── profile.a
├── pprof.a
├── race.a
└── trace.a
pkg/linux_amd64/runtime.a [error opening dir]
pkg/linux_amd64/net/http
├── cgi.a
├── cookiejar.a
├── fcgi.a
├── httptest.a
├── httptrace.a
├── httputil.a
├── internal.a
└── pprof.a
pkg/linux_amd64/net/http.a [error opening dir]
pkg/linux_amd64/vendor/golang_org/x/net/http
├── httpguts.a
└── httpproxy.a
pkg/linux_amd64/vendor/golang_org/x/net/http2
└── hpack.a

3 directories, 19 files
@oec

This comment has been minimized.

Copy link

@oec oec commented Oct 26, 2018

Maybe this is unrelated to the Alpine image, as I experience the same issue with go 1.11.1 on Ubuntu 18.04.1 LTS, using vanilla go downloaded from https://dl.google.com/go/go1.11.1.linux-amd64.tar.gz.

stevendanna added a commit to chef/scaffolding-go that referenced this issue Nov 16, 2018
This unpins our version of core/scaffolding-go which in turn will pull
in the latest version of core/go.

All of A2 builds and passes tests with this change in place:

https://buildkite.com/chef/chef-a2-master-verify/builds/17409

In the past we had pinned our version of Go to avoid this bug:

golang/go#28065

However, rather than pinning, we should track down any places where
that is still an issue and either make sure those builds either have
gcc available or set GCO_ENABLED=0.

Signed-off-by: Steven Danna <steve@chef.io>
@ArtoriaRen

This comment has been minimized.

Copy link

@ArtoriaRen ArtoriaRen commented Nov 21, 2018

I am on Ubuntu 18.04 and encountered the same error :
runtime/cgo
exec: "gcc": executable file not found in $PATH

when I run "go test".

Solution: sudo apt install gcc

stevendanna added a commit to chef/scaffolding-go that referenced this issue Nov 27, 2018
This unpins our version of core/scaffolding-go which in turn will pull
in the latest version of core/go.

All of A2 builds and passes tests with this change in place:

https://buildkite.com/chef/chef-a2-master-verify/builds/17409

In the past we had pinned our version of Go to avoid this bug:

golang/go#28065

However, rather than pinning, we should track down any places where
that is still an issue and either make sure those builds either have
gcc available or set GCO_ENABLED=0.

Signed-off-by: Steven Danna <steve@chef.io>
@thepudds

This comment has been minimized.

Copy link

@thepudds thepudds commented Nov 29, 2018

Hi @mvdan (and/or anyone else on this issue)

Good chance you already saw this, but it seems #26988 is now hopefully fixed in tip (where this issue #28065 was originally spun out from #26988).

I haven't personally seen anything that explicitly suggests that this issue #28065 might also be addressed, but it could be worth a quick try off of tip to see the situation here has improved.

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Nov 29, 2018

Thanks - I'll build tip on Alpine, and see if this bug is still present.

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Nov 29, 2018

Still appears to happen. New script:

docker run -i golang:1.11.2-alpine3.8 sh <<SCRIPT

cd /
apk add --no-cache --virtual .build-deps bash gcc musl-dev openssl git
go version
export GOROOT_BOOTSTRAP="$(go env GOROOT)"
git clone --depth=1 https://go.googlesource.com/go tip
cd tip/src
./make.bash
apk del .build-deps
/tip/bin/go version

cd /go
mkdir -p src/foo.com/bar
cd src/foo.com/bar

cat <<EOF >main.go
package main

import _ "net/http"

func main() {}
EOF

/tip/bin/go test

SCRIPT

Output:

[...]
go version devel +ba2e8f6 Thu Nov 29 18:18:51 2018 +0000 linux/amd64
# runtime/cgo
exec: "gcc": executable file not found in $PATH
@bcmills bcmills added the GoCommand label Nov 29, 2018
@bcmills bcmills changed the title cmd/go: simple 'go test' on 1.11 requires gcc in GOPATH mode cmd/go: 'go test' on alpine requires gcc in GOPATH mode Dec 7, 2018
@andybons andybons modified the milestones: Go1.12, Go1.13 Feb 12, 2019
@jonicreide

This comment has been minimized.

Copy link

@jonicreide jonicreide commented Feb 22, 2019

Hey guys, I think that you can include this short command in yours Dockerfile

ENV CGO_ENABLED=0

@mvdan

This comment has been minimized.

Copy link
Member Author

@mvdan mvdan commented Feb 23, 2019

@MatheusViniciusAndrade please read the previous comments - CGO_ENABLED is already mentioned multiple times.

charleskorn added a commit to charleskorn/batect-sample-golang that referenced this issue Mar 16, 2019
This is a little less confusing and improves IDE support.

This required changing to a non-Alpine-based image for the build env due
to golang/go#28065, and therefore building a
statically-linked binary for the production image.
bzz added a commit to bzz/enry that referenced this issue Apr 9, 2019
With go1.11 `go test` in GOPATH mode somehow
seems to depend on GCC. See golang/go#28065

This change only enables cgo for CI profiles that
need it. Those are the ones that seem to fail
on TravisCI now, presumably due to some compiler
version missmatch.

That is a workaround and does not happen in GO11MODULE mode.

Signed-off-by: Alexander Bezzubov <bzz@apache.org>
bzz added a commit to bzz/enry that referenced this issue Apr 9, 2019
With go1.11 `go test` in GOPATH mode somehow
seems to depend on GCC. See golang/go#28065

This change only enables cgo for CI profiles that
need it. Those are the ones that seem to fail
on TravisCI now, presumably due to some compiler
version missmatch.

That is a workaround and does not happen in GO11MODULE mode.

Signed-off-by: Alexander Bezzubov <bzz@apache.org>
bzz added a commit to bzz/enry that referenced this issue Apr 9, 2019
With go1.11 `go test` in GOPATH mode somehow
seems to depend on GCC. See golang/go#28065

This change only enables cgo for CI profiles that
need it. Those are the ones that seem to fail
on TravisCI now, presumably due to some compiler
version missmatch.

That is a workaround and does not happen in GO11MODULE mode.

Signed-off-by: Alexander Bezzubov <bzz@apache.org>
bzz added a commit to bzz/enry that referenced this issue Apr 9, 2019
With go1.11 `go test` in GOPATH mode somehow
seems to depend on GCC. See golang/go#28065

This change only enables cgo for CI profiles that
need it. Those are the ones that seem to fail
on TravisCI now, presumably due to some compiler
version missmatch.

That is a workaround and does not happen in GO11MODULE mode.

Signed-off-by: Alexander Bezzubov <bzz@apache.org>
bzz added a commit to bzz/enry that referenced this issue Apr 11, 2019
With go1.11 `go test` in GOPATH mode somehow
seems to depend on GCC. See golang/go#28065

This change only enables cgo for CI profiles that
need it. Those are the ones that seem to fail
on TravisCI now, presumably due to some compiler
version missmatch.

That is a workaround and does not happen in GO11MODULE mode.

Signed-off-by: Alexander Bezzubov <bzz@apache.org>
@bcmills bcmills modified the milestones: Go1.13, Unplanned May 7, 2019
@dennwc dennwc referenced this issue May 22, 2019
@bcmills bcmills changed the title cmd/go: 'go test' on alpine requires gcc in GOPATH mode cmd/go: 'go test' on alpine and ubuntu requires gcc in GOPATH mode May 28, 2019
@dackroyd

This comment has been minimized.

Copy link

@dackroyd dackroyd commented Jul 10, 2019

I have been experiencing this issue in attempting to upgrade from Go 1.10 with my application build (not using Go modules), which runs in Alpine docker. As others are experiencing, go test fails with the dreaded:

# runtime/cgo
exec: "gcc": executable file not found in $PATH

Looking through the changes for Go 1.11 (where this issue originally surfaces), I found that one of those changes is that when vetting test packages, they are 'rebuilt as needed': 804d032#diff-acaf53a9cd478507ebbcf85037940b4d

Disabling vet during go test proved to avoid the issue for my case, i.e. go test -v -vet=off ./...

As part of our Go builds, we already run vet along with other static checks, and we also run tests with the race detector enabled in a 'stretch' image (which will still have vet enabled), so I don't think we've lost any checks by doing this.

@JAicewizard

This comment has been minimized.

Copy link

@JAicewizard JAicewizard commented Aug 6, 2019

Is this still being worked on?
Most related issues have been closed as of now.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Aug 6, 2019

@JAicewizard, this issue is not assigned to anyone and is milestoned to Unplanned. If someone wants to investigate and send a fix that'd be great, but at the moment the impact seems small enough that other issues are higher priority.

@quocanh1897

This comment has been minimized.

Copy link

@quocanh1897 quocanh1897 commented Sep 19, 2019

@dackroyd thank you very much, it works like a charm!

@eandre

This comment has been minimized.

Copy link

@eandre eandre commented Oct 29, 2019

I was looking into this briefly and came across the build log and script for the Go package for alpine linux:

Build log: https://build.alpinelinux.org/buildlogs/build-edge-x86/community/go/go-1.13.3-r0.log
Build script: https://git.alpinelinux.org/aports/tree/community/go/APKBUILD

It seems to me this goes through a lot of steps that are likely to break the build cache. It ignores a lot of so-called "broken tests" related to runtime and cgo. The build log has a lot of errors related to not being able to find runtime/cgo. A lot of compiled pkg objects are deleted at the end of the build.

It's not clear to me why all these steps are being taken. Maybe somebody with more insight into the Go build process could share which part(s) of these build steps are likely to break the build cache for go test? And if someone from Alpine could shed some light on why these steps are being taken that would be nice.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 29, 2019

@eandre Happy to answer specific questions but in general I do not know why the Alpine build script is making those choices. I think you need to ask the author of the script.

@eandre

This comment has been minimized.

Copy link

@eandre eandre commented Oct 29, 2019

@ianlancetaylor I guess the main question is why -vet=on (whether passed explicitly or by default) to go test causes runtime/cgo to be rebuilt?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 29, 2019

I don't know. Is the problem easy to reproduce? Does it happen every time? Can you run GODEBUG=gocachehash=1 go test -x and attach the output?

@eandre

This comment has been minimized.

Copy link

@eandre eandre commented Oct 29, 2019

Sure, here's the output of that command: https://gist.github.com/eandre/da1fc79f1fe78179869583a80c9d4fcb

Yes, it happens every time.

@eandre

This comment has been minimized.

Copy link

@eandre eandre commented Oct 29, 2019

Here's GODEBUG=gocachehash=1 go test -vet=off -x for comparison: https://gist.github.com/eandre/d71a237df88d11ba32a6eefbcaeaf11a

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 30, 2019

Thanks. The hash is the same, so that wasn't it.

I think the problem is that vet wants to collect information, known as "facts", for runtime/cgo (as well as all the other packages that the problem depends on). That information can be cached, but it hasn't been cached on your system. For a package that uses cgo, vet needs to run cgo to see the generated files. So vet invokes cgo. But on your system, which does not have "gcc" installed, that fails.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.