Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: aix-ppc64 builder has non-reproducible cgo builds #56896

Open
bcmills opened this issue Nov 22, 2022 · 5 comments
Open

x/build: aix-ppc64 builder has non-reproducible cgo builds #56896

bcmills opened this issue Nov 22, 2022 · 5 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-AIX
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Nov 22, 2022

What version of Go are you using (go version)?

~/go-review/src$ go version
go version devel go1.20-fe7d3a0df5 Mon Nov 21 19:48:00 2022 -0500 linux/amd64

~/go-review/src$ mote run gcc --version
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote363460504/bcmills-aix-ppc64-0.stdout"...
gcc (GCC) 8.4.0

What did you do?

Ran go list -export -json=BuildID runtime/cgo before and after clearing the Go build cache.

~/go-review/src/cmd/dist$ mote run ./go/bin/go list -export -json=BuildID runtime/cgo
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote3714306807/bcmills-aix-ppc64-0.stdout"...
{
        "BuildID": "ogJ-10b1BIf2OIMT5g5K/4jyhoIAusu99uwtt-kuy"
}
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote3714306807/bcmills-aix-ppc64-0.stdout".

~/go-review/src/cmd/dist$ mote run ./go/bin/go clean -cache
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote225234382/bcmills-aix-ppc64-0.stdout"...
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote225234382/bcmills-aix-ppc64-0.stdout".

~/go-review/src/cmd/dist$ mote run ./go/bin/go list -export -json=BuildID runtime/cgo
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote3775804530/bcmills-aix-ppc64-0.stdout"...
{
        "BuildID": "ogJ-10b1BIf2OIMT5g5K/7hDfzCuKuu_KQFpt3mXk"
}
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote3775804530/bcmills-aix-ppc64-0.stdout".

Note that the first part of the build ID (describing the inputs to the build) is identical; however, the second part (describing the output of the compiler) is not.

What did you expect to see?

Because the runtime/cgo package is within GOROOT, its build should always be reproducible (see #50183), and so each fresh rebuild of the package should result in the same BuildID, and any package in GOROOT that uses it (such as net) should also have a reproducible build.

Setting CGO_ENABLED=0 does result in a reproducible build for the net package on the aix-ppc64 builder:

~/go-review/src/cmd/dist$ gomote run -e 'CGO_ENABLED=0' bcmills-aix-ppc64-0 ./go/bin/go list -export -json=BuildID net
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote3253411634/bcmills-aix-ppc64-0.stdout"...
{
        "BuildID": "_EPUfZVuKfZHEmEBc02h/7QWWxSr9bui2XajXR10N"
}
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote3253411634/bcmills-aix-ppc64-0.stdout".

~/go-review/src/cmd/dist$ gomote run -e 'CGO_ENABLED=0' bcmills-aix-ppc64-0 ./go/bin/go clean -cache
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote3293010067/bcmills-aix-ppc64-0.stdout"...
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote3293010067/bcmills-aix-ppc64-0.stdout".

~/go-review/src/cmd/dist$ gomote run -e 'CGO_ENABLED=0' bcmills-aix-ppc64-0 ./go/bin/go list -export -json=BuildID net
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote1809113884/bcmills-aix-ppc64-0.stdout"...
{
        "BuildID": "_EPUfZVuKfZHEmEBc02h/7QWWxSr9bui2XajXR10N"
}
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote1809113884/bcmills-aix-ppc64-0.stdout".

And on linux/amd64 (and presumably other platforms), we do see reproducible builds for runtime/cgo even with cgo enabled:

~/go-review/src$ go list -export -json=BuildID runtime/cgo
{
        "BuildID": "WhJYk_P3Gk345k1UDp2X/iflidhKq7bnW0242XAPO"
}

~/go-review/src$ go clean -cache

~/go-review/src$ go list -export -json=BuildID runtime/cgo
{
        "BuildID": "WhJYk_P3Gk345k1UDp2X/iflidhKq7bnW0242XAPO"
}

What did you see instead?

With cgo enabled, the BuildID for the runtime/cgo and net packages on the aix-ppc64 builder differ each time the package is rebuilt from a clean cache.

(attn @golang/aix; CC @ianlancetaylor in case he has any tips on cgo reproducibility)

@bcmills bcmills added the OS-AIX label Nov 22, 2022
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 22, 2022
@gopherbot gopherbot added this to the Unreleased milestone Nov 22, 2022
@gopherbot
Copy link

gopherbot commented Nov 22, 2022

Change https://go.dev/cl/452680 mentions this issue: cmd/dist: skip the staleness check for 'cmd' when testing on aix-ppc64

@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 22, 2022
gopherbot pushed a commit that referenced this issue Nov 22, 2022
The gcc toolchain on the aix-ppc64 builder apparently does not achieve
reproducible builds for packages that use cgo, which causes the
binaries in cmd that use package "net" (cmd/go, cmd/pprof, and
cmd/trace) to appear stale whenever the Go build cache is cleared.

For now, we work around the staleness by rebuilding std and simply not
checking whether cmd is stale.

For #56896.
Updates #47257.

Change-Id: I15f86e72dee53904b881710d5d5d613872361510
Reviewed-on: https://go-review.googlesource.com/c/go/+/452680
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Ayappan Perumal <ayappanec@gmail.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

gopherbot commented Nov 22, 2022

Change https://go.dev/cl/452775 mentions this issue: cmd/dist: skip rebuilding packages during builder testing

@bcmills bcmills changed the title x/build: aix-ppc64 builder failing cmd/dist staleness checks at CL 452457 due to non-reproducible cgo builds x/build: aix-ppc64 builder has non-reproducible cgo builds Nov 22, 2022
@bcmills
Copy link
Member Author

bcmills commented Nov 22, 2022

The builder is once again passing (by skipping the staleness checks), so I've retitled the issue to reflect the underlying reproducibility problem that remains.

@laboger
Copy link
Contributor

laboger commented Nov 23, 2022

@ayappanec Would you look into this?

@ayappanec
Copy link

ayappanec commented Nov 24, 2022

Yes, Looking into it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-AIX
Projects
None yet
Development

No branches or pull requests

5 participants