Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: hub+kubernetes has random cgo_gotypes build path #63851

Closed
bmwiedemann opened this issue Oct 31, 2023 · 14 comments
Closed

cmd/go: hub+kubernetes has random cgo_gotypes build path #63851

bmwiedemann opened this issue Oct 31, 2023 · 14 comments
Labels

Comments

@bmwiedemann
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.21.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/abuild/.cache/go-build'
GOENV='/home/abuild/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/abuild/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/abuild/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib64/go/1.21'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/lib64/go/1.21/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.3'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2297302055=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I did a double-build of hub and kubernetes1.27 in openSUSE.

# on Debian or openSUSE
osc co openSUSE:Factory/hub ; cd $_
for N in 1 2 ; do
    osc build --clean --nopreinstallimage --keep-pkg=RPMS.$N --noservice standard
    unrpm RPMS.$N/hub-2*.x86_64.rpm
    strings usr/bin/hub > $N.strings
done
diff -u {1,2}.strings

What did you expect to see?

two identical results

What did you see instead?

strings of the produced hub binary varied thusly:

 /usr/lib64/go/1.21/src/net/tcpsockopt_unix.go
 /usr/lib64/go/1.21/src/net/udpsock_posix.go
 /usr/lib64/go/1.21/src/net/unixsock_posix.go
-/tmp/go-build1489893222/b079/_cgo_gotypes.go
+/tmp/go-build4283137480/b079/_cgo_gotypes.go
 /usr/lib64/go/1.21/src/net/cgo_unix_cgo.go
 /usr/lib64/go/1.21/src/crypto/des/block.go

This was initially reported in kubernetes/kubernetes#110928

@bcmills
Copy link
Member

bcmills commented Oct 31, 2023

osc build is not part of the Go project. Can you reproduce this problem using go build?

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. GoCommand cmd/go labels Oct 31, 2023
@bcmills bcmills changed the title affected/package: hub+kubernetes has random cgo_gotypes build path cmd/go: hub+kubernetes has random cgo_gotypes build path Oct 31, 2023
@bmwiedemann
Copy link
Author

bmwiedemann commented Nov 4, 2023

I made a more simplified reproducer with hub-2.14.2:

for i in 1 2 ; do 
  rm -rf ~/.cache/go-build
  go build -mod=vendor -gcflags 'all=-trimpath' -o bin/hub
  md5sum bin/hub
done

Both the -trimpath and removal of ~/.cache/go-build are required to trigger the issue.

@seankhliao
Copy link
Member

Why are you using -gcflags=all=-trimpath instead of the -trimpath build flag?

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Nov 10, 2023
@bmwiedemann
Copy link
Author

also kubernetes/kubernetes@bf27cad has some insight.

@bmwiedemann
Copy link
Author

I did a quick re-test and both

for i in 1 2 ; do rm -rf ~/.cache/go-build ; go build -mod=vendor -gcflags '-trimpath' -o bin/hub && md5sum bin/hub ; done

and

for i in 1 2 ; do rm -rf ~/.cache/go-build ; go build -mod=vendor '-trimpath' -o bin/hub && md5sum bin/hub ; done

would produce bit-identical results.

What does the all= do to not cover the random /tmp/go-build4283137480/b079/_cgo_gotypes.go path?

@bcmills
Copy link
Member

bcmills commented Nov 10, 2023

@bmwiedemann, the -gcflags flag specifies flags to be passed to the compiler (cmd/compile, formerly known as gc).

Those flags necessarily do not affect the output of the C compiler or linker or the cgo command. One of those is likely where that path is coming from.

In general the -gcflags flags are “you break it, you bought it”. To build reproducible binaries using go build, you need to pass the -trimpath flag to the go command instead.

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Nov 10, 2023
@bmwiedemann
Copy link
Author

bmwiedemann commented Nov 11, 2023

Those flags necessarily do not affect the output of the C compiler or linker or the cgo command. One of those is likely where that path is coming from.

For me, this does not yet explain, why this helps for reproducibility:

- -gcflags 'all=-trimpath'
+ -gcflags '-trimpath'

We are still only passing it to the go-compiler, so this issue should not be from the C compiler or linker?

If we assume, only go -trimpath is the right way,
for hub the patch is simple, but for kubernetes rather not so much. Maybe you could work with them on kubernetes/kubernetes#110928?

bmwiedemann added a commit to bmwiedemann/hub that referenced this issue Nov 11, 2023
per golang/go#63851 this is the recommended way
to achieve reproducible builds.

This patch was done while working on reproducible builds for openSUSE.
@bmwiedemann
Copy link
Author

Another observation: just by dropping the all=-trimpath it becomes reproducible (when re-building in the same path).
This seems to contradict your explanation.

@seankhliao
Copy link
Member

closing as not an issue in the go toolchain

@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Nov 11, 2023
@bcmills
Copy link
Member

bcmills commented Nov 13, 2023

@bmwiedemann, the -trimpath flag supported by cmd/compile requires a prefix argument (see https://pkg.go.dev/cmd/compile). So probably setting -gcflags 'all=-trimpath' is actually undoing whatever -trimpath effects cmd/go itself is actually setting on the command line for the go build invocation.

At any rate, the conclusion here is the same either way: if you want reproducible binaries, you need to pass the -trimpath flag to cmd/go itself, not to the compiler via -gcflags.

@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 13, 2023
@thockin
Copy link

thockin commented Nov 15, 2023

I started digging here, and it seems like -trimpath should be much simpler BUT... the semantic is different.

-gcflags 'all=-trimpath=$(pwd)' converts "/home/thockin/src/kube/pkg/foo/bar.go" into "pkg/foo/bar.go", but -trimpath produces "k8s.io/kubernetes/pkg/foo/bar.go". One of these is file path, one is a module. So now there are tools which fail to find the file.

Is there a simple way to convert a module path to a file path ? It looks like -trimpath doesn't take an arg.

@bcmills
Copy link
Member

bcmills commented Nov 15, 2023

@thockin, binaries built with -trimpath from modules fundamentally cannot embed file paths for modules. The file paths would have to be relative, but there is no one consistent thing they could all be relative to. (It can't be “the module's root directory”, because not all inputs to the build are found in the same module — some of them are from $(go env GOROOT)/src, and in the general case some are from various subdirectories of $(go env GOMODCACHE).)

Instead, -trimpath records the paths as the package import path followed by the filename within the package directory.

You can get the file path for a give package import path using go list -json=ImportPath,Dir, although bear in mind that the ImportPath field is unfortunately overloaded to include more than just the import path. (I can't find a proposal to fix that at the moment, so I'm filing a new one now.)

@thockin
Copy link

thockin commented Nov 15, 2023

It's unfortunate - we vendor everything, so we COULD resolve to file paths. It works the way we want (mostly) with -gcflags.

So right now I am hunting down the places where we get filenames which (after this change) are not actual paths, and seeing if/how to fix those.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants