cmd/go: non reproducible builds since cgo_ldflag that contains build path

[sandy-lcq]

Go version

1.25.3

Output of go env in your module/workspace:

GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build/.cache'
GOCACHEPROG=''
GODEBUG='gocachehash=1'
GOENV=''
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot -I . -fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/go-build-tmp/go-build3678307125=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTMPDIR='/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/go-build-tmp'
GOTOOLCHAIN='auto'
GOTOOLDIR='/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.3'
GOWORK=''

What did you do?

I am compiling buildah with yocto project, under build path ra and build path rb-extend, here is the build command:

x86_64-wrs-linux-go install -v -ldflags="-r /usr/lib64/go/pkg/linux_amd64_dynlink -I /usr/lib64/ld-linux-x86-64.so.2 -extldflags ' -m64 -march=x86-64-v3 -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot -Wl,-rpath-link=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot/usr/lib64/go/pkg/linux_amd64_dynlink -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5=/usr/src/debug/buildah/1.41.5 -ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build=/usr/src/debug/buildah/1.41.5 -ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot= -ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native= -Wl,-z,relro,-z,now'" -trimpath -mod vendor -buildmode=pie ./cmd/buildah

With above command, the generated buildah is not reproducible,
BUT, after adding "-buildid=", the generated buildah is reproducible.

go buildid make the binary buildah not reproducible.

maybe this is related to: https://github.com/golang/go/blob/master/src/cmd/go/internal/modindex/read.go#L70

the buildid include hash of module root directory, in my case, module root directory is different, so maybe the TODO in comments,
use module path instead of module root directory can resolve this problem.

Here is the log under build path ra:

HASH[moduleIndex]: "go1.25.3"
HASH[moduleIndex]: "modroot /ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5/src/github.com/containers/buildah\n"
HASH[moduleIndex]: "package go1.25.3 go index v2 /ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5/src/github.com/containers/buildah/cmd/buildah\n"
...

Here is the log under build path rb-extend:

HASH[moduleIndex]: "go1.25.3"
HASH[moduleIndex]: "modroot /rb-extend/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5/src/github.com/containers/buildah\n"
HASH[moduleIndex]: "package go1.25.3 go index v2 /rb-extend/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5/src/github.com/containers/buildah/cmd/buildah\n"
...

What did you see happen?

I am compiling buildah with yocto project, under build path ra and build path rb-extend, here is the build command:

x86_64-wrs-linux-go install -v -ldflags="-r /usr/lib64/go/pkg/linux_amd64_dynlink -I /usr/lib64/ld-linux-x86-64.so.2 -extldflags ' -m64 -march=x86-64-v3 -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot -Wl,-rpath-link=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot/usr/lib64/go/pkg/linux_amd64_dynlink -Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5=/usr/src/debug/buildah/1.41.5 -ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build=/usr/src/debug/buildah/1.41.5 -ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot= -ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/recipe-sysroot-native= -Wl,-z,relro,-z,now'" -trimpath -mod vendor -buildmode=pie ./cmd/buildah

With above command, the generated buildah is not reproducible, I tried to add "-buildvcs=false ", not work.
BUT, after adding "-buildid=", the generated buildah is reproducible. Seems there is still something make buildid not reproducile.

I checked the compile log and output of "go tool buildid ./buildah", I can see some hash difference, but I cannot get what make this happen? Could you provide some guide, so that I can collect more info for debuging this? Thanks.``

What did you expect to see?

the binary buildah compiled out by go is reproducible

[gabyhelp]

Related Issues

• cmd/go: not reproducible buildid when building with -trimpath #76743 (closed)
• cmd/go: build ID depends on the full path of Go files passed on the command line #48557 (closed)
• cmd/go: go buildid actionID part is not reproducible with clean cache #59553 (closed)
• cmd/go: support reproducible buildid when building with -trimpath #34186 (closed)
• cmd/go: executables built with `-trimpath` are not reproducible with development Go toolchains unless `-ldflags` removes `-buildid=` #59525 (closed)
• build id varies from parallelism #42159 (closed)
• cmd/link: Mismatch between hashed and actual linker config prevents reproducible builds #38989 (closed)
• cmd/go: value of -buildmode in buildinfo can differ from what is used in build ID #63559
• debug/buildinfo/testdata: test binaries should be more reproducible #71734

Related Discussions

• Reproducible builds and rewriting buildid in binary

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[JunyangShao]

@golang/command-line

[sandy-lcq]

Hi,

After do more investigation, it turns out that the not reproducible is not related to the moduleIndex. But related to the cgo_ldflag in _go_.o, and this _go_.o is one of file for ar archive file _pkg_.a. This cgo_ldflag contains build path, so cause not reproducible.

Take example:
In my env, build runtime/cgo will generate _pkg_.a, and _go_.o in _pkg_.a contains contents like below:

["cgo_ldflag","-ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/sources/buildah-1.41.5=/usr/src/debug/buildah/1.41.5"],["cgo_ldflag","-ffile-prefix-map=/ra/build/tmp/work/x86-64-v3-wrs-linux/buildah/1.41.5/build=/usr/src/debug/buildah/1.41.5"]

So when call buildid.FindAndHash, the content hash of _pkg_.a will be calculated, and the build path make it not reproducible. And all the package deps on it will not reproducible.

Besides build path, in _go_.o and __.PKGDEF, 'build id "actionid/contentid" also different since above build path diff

I find that there is function like excludeHostBuildID, try to remove content that influence reproducible before calculate content hash:
https://github.com/golang/go/blob/master/src/cmd/internal/buildid/rewrite.go#L46

So I tried to exclude the buildpath and buildid before calculate content hash, but seems this content hash cannot be changed like this:

It will cause errors like "go: internal cache error: cache verify failed" or "Stale: xxx build id mismatch", I have no idea how the content hash is handle in the whole process, so I am stucked and have no idea how this should be fixed.

Could experts in this please help to review and give your comments, Thanks.