Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: cgo-enabled binary built with Ubuntu 22.04 is incompatible with Ubuntu 20.04 #57328

Closed
zeisss opened this issue Dec 15, 2022 · 9 comments
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Unfortunate
Milestone

Comments

@zeisss
Copy link

zeisss commented Dec 15, 2022

What version of Go are you using (go version)?

$ ~/go/bin/go version
go version go1.19.4 linux/amd64

Does this issue reproduce with the latest release?

This is the latest stable release from the website.

What operating system and processor architecture are you using (go env)?

go env Output
$ ~/go/bin/go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ubuntu/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ubuntu/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build646395045=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Building a minimal binary (using net/http code) with -trimpath on Ubuntu 22.04 makes it incompatible with Ubuntu 20.04:

ubuntu@jammy:~$ cat main.go
package main

import "fmt"
import "net/http"

func main() {
	fmt.Println("Hello World")

	http.ListenAndServe(":8080", nil)
}
ubuntu@jammy:~$ ~/go/bin/go build -o main-jammy -trimpath ./main.go

Copying the resulting binary to an Ubuntu 20.04 machine yields the following error:

$ ./main-jammy
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./main-jammy)
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./main-jammy)

Dropping -trimpath from the go build call and the binary works.

What did you expect to see?

Invoking main-jammy on an Ubuntu 20.04 machine should yield Hello World and start a webserver.

What did you see instead?

$ ./main-jammy
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./main-jammy)
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./main-jammy)

Context

Github Actions is currently rolling out Ubuntu 22.04 for its ubuntu-latest tag and as a result some of our releases are becoming invalid with Ubuntu 20.0.

@zeisss zeisss changed the title affected/package: cmd/go: -trimpath makes binary incompatible with Ubuntu 20.04 when build on Ubuntu 22.04 Dec 15, 2022
@bcmills
Copy link
Member

bcmills commented Dec 15, 2022

go version go1.19.4 linux/amd64

Is this a regression from a previous release? (Did it ever work?)

@zeisss
Copy link
Author

zeisss commented Dec 15, 2022

Previously releases have been build on Ubuntu 20.04 on Github Actions (we are using the ubuntu-latest tag), so I cannot answer that. If I find time tomorrow, I might be able to test this with older releases.

@thanm thanm added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Dec 15, 2022
@thanm
Copy link
Contributor

thanm commented Dec 15, 2022

@golang/compiler

@bcmills
Copy link
Member

bcmills commented Dec 15, 2022

I think -trimpath is something of a red herring. Here's my hypothesis:

  • The problem is that your installed glibc has a symbol definition that is newer than the glibc of the machine on which you are deploying the binary.
  • When you build without -trimpath, the problem is masked because you link against the precompiled .a files that shipped with go1.19.4, and link against the glibc symbols that happened to be used to build that distribution's cgo dependencies.
  • However, when you build with -trimpath, the cache key for the precompiled files no longer matches, and you get a rebuild with your installed glibc.

We can bust the cache in other ways, too (such as by setting CGO_CFLAGS), reproducing the same symptom:

$ go1.19.4 build -o main main.go

$ nm -D main | grep pthread_attr_getstacksize
                 U pthread_attr_getstacksize@GLIBC_2.2.5

$ go1.19.4 build -trimpath -o main main.go

$ nm -D main | grep pthread_attr_getstacksize
                 U pthread_attr_getstacksize@GLIBC_2.34

$ CGO_CFLAGS='-O2 -g -D__go_unused_variable=1' go1.19.4 build -o main main.go

$ nm -D main | grep pthread_attr_getstacksize
                 U pthread_attr_getstacksize@GLIBC_2.34

@bcmills
Copy link
Member

bcmills commented Dec 15, 2022

And in Go 1.20, you will see the newer symbols on all builds because we're not shipping precompiled archives at all (#47257). 😅

@bcmills
Copy link
Member

bcmills commented Dec 15, 2022

The workarounds I can see for your use-case are:

  • Build the program on the system with the older glibc (so that it uses the older symbols).
  • Set CGO_ENABLED=0 to build a binary that doesn't depend on glibc at all.
  • Set various flags (see cmd/go: build: add -static flag #26492) to build a fully-static cgo-enabled binary.

@bcmills
Copy link
Member

bcmills commented Dec 15, 2022

But, I am left a little bit puzzled about one thing: I don't expect -trimpath to actually affect the cache keys for the standard library: as of CL 380915 we should be building standard-library packages as if -trimpath were always set. 🤔

I'll file a separate issue to look into that.

@bcmills bcmills changed the title cmd/go: -trimpath makes binary incompatible with Ubuntu 20.04 when build on Ubuntu 22.04 cmd/go: cgo-enabled binary built with Ubuntu 22.04 is incompatible with Ubuntu 20.04 Dec 15, 2022
@bcmills bcmills added the GoCommand cmd/go label Dec 15, 2022
@bcmills bcmills added this to the Unplanned milestone Dec 15, 2022
@bcmills
Copy link
Member

bcmills commented Dec 15, 2022

I've filed #57346 for the cache-invalidation behavior. Otherwise, I don't think there is anything we can feasibly do for this on the cmd/go side: cgo-enabled Go binaries inherently have many of the same compatibility issues as ordinary C binaries.

@bcmills bcmills closed this as not planned Won't fix, can't repro, duplicate, stale Dec 15, 2022
@zeisss
Copy link
Author

zeisss commented Dec 16, 2022

Build the program on the system with the older glibc (so that it uses the older symbols).

Yeah, we will go with that. Since we control most execution environments, this is doable and easiest. I mainly reported this since this was surprising to me and non-obvious from the -trimpath description.

Thanks for looking into this so quickly!

g0rbe added a commit to elmasy-com/columbus-server that referenced this issue Mar 23, 2023
g0rbe added a commit to elmasy-com/columbus-scanner that referenced this issue Mar 31, 2023
Tal-or added a commit to openshift-kni/mixed-cpu-node-plugin that referenced this issue Jun 12, 2023
Due to recent changes, the plugin image build has failed.
The following changes has been added:
1. Fixing the command name in the DaemonSet spec
2. Pin golang to 1.20
3. Running the go build command with the right arguments
4. changing CGO_ENABLED=0 - The binary failed to execute
and complains about GLIBC_2.32 not found.
Running with CGO_ENABLED=0 build a binary that doesn't depend on glibc:
golang/go#57328 (comment)

Signed-off-by: Talor Itzhak <titzhak@redhat.com>
Tal-or added a commit to openshift-kni/mixed-cpu-node-plugin that referenced this issue Jun 13, 2023
Due to recent changes, the plugin image build has failed.
The following changes has been added:
1. Fixing the command name in the DaemonSet spec
2. Pin golang to 1.20
3. Running the go build command with the right arguments
4. changing CGO_ENABLED=0 - The binary failed to execute
and complains about GLIBC_2.32 not found.
Running with CGO_ENABLED=0 build a binary that doesn't depend on glibc:
golang/go#57328 (comment)

Signed-off-by: Talor Itzhak <titzhak@redhat.com>
kirederik added a commit to syntasso/kratix that referenced this issue Jun 15, 2023
1.19.10 includes a change on linked libraries that's messing up our manager binary

see golang/go#47257 and golang/go#57328
(issues are closed, but the error is the same)
wkalt added a commit to foxglove/foxglove-cli that referenced this issue Sep 27, 2023
Due to golang/go#57328, binaries built on
Github's "ubuntu-latest" don't run on Ubuntu 20.04. Setting
CGO_ENABLED=0 at build fixes this. Since we don't rely on CGO in this
repo, this seems like the easiest fix.
wkalt added a commit to foxglove/foxglove-cli that referenced this issue Sep 27, 2023
By default we are relying on cgo for DNS resolution, which causes
binaries built on ubuntu 22.04 to break on ubuntu 20.04 due to some
changes in glibc. See golang/go#57328

This changes us to not rely on that, which causes builds to succeed.
wkalt added a commit to foxglove/foxglove-cli that referenced this issue Sep 27, 2023
By default we are relying on cgo for DNS resolution, which causes
binaries built on ubuntu 22.04 to break on ubuntu 20.04 due to some
changes in glibc. See golang/go#57328

This changes us to not rely on that, which causes builds to succeed.
mikeb26 added a commit to mikeb26/gptcli that referenced this issue Nov 8, 2023
The CircleCI go build images' toolchain depends on a version of GLIBC
that is higher than the version that ships on Ubuntu 20.04. This means
users trying to install the released binaries on Ubuntu 20.04 will
see:

```
$ gptcli
gptcli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not
  found (required by gptcli)

gptcli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not
  found (required by gptcli)
```

This commit attempts to mitigate the problem by building w/
CGO_ENABLED=0 as suggested by:
  golang/go#57328
mikeb26 added a commit to mikeb26/bashgpt that referenced this issue Nov 8, 2023
The CircleCI go build images' toolchain depends on a version of GLIBC
that is higher than the version that ships on Ubuntu 20.04. This means
users trying to install the released binaries on Ubuntu 20.04 will
see:

```
$ bashgpt
bashgpt: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not
  found (required by bashgpt)
bashgpt: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not
  found (required by bashgpt)
```

This commit attempts to mitigate the problem by building w/
CGO_ENABLED=0 as suggested by:
golang/go#57328
mikeb26 added a commit to mikeb26/spotsh that referenced this issue Nov 8, 2023
The CircleCI go build images' toolchain depends on a version of GLIBC
that is higher than the version that ships on Ubuntu 20.04. This means
users trying to install the released binaries on Ubuntu 20.04 will
see:

```
$ spotsh
spotsh: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not
  found (required by spotsh)
spotsh: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not
  found (required by spotsh)
```

This commit attempts to mitigate the problem by building w/
CGO_ENABLED=0 as suggested by:
golang/go#57328
macno added a commit to theoapp/theo-agent that referenced this issue Dec 3, 2023
macno added a commit to theoapp/theo-agent that referenced this issue Dec 3, 2023
macno added a commit to theoapp/theo-agent that referenced this issue Dec 3, 2023
macno added a commit to theoapp/theo-agent that referenced this issue Dec 3, 2023
@golang golang locked and limited conversation to collaborators Dec 16, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. Unfortunate
Projects
None yet
Development

No branches or pull requests

4 participants