Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: aix-ppc64 builder has non-reproducible cgo builds #56896

Open
bcmills opened this issue Nov 22, 2022 · 10 comments
Open

x/build: aix-ppc64 builder has non-reproducible cgo builds #56896

bcmills opened this issue Nov 22, 2022 · 10 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-AIX
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Nov 22, 2022

What version of Go are you using (go version)?

~/go-review/src$ go version
go version devel go1.20-fe7d3a0df5 Mon Nov 21 19:48:00 2022 -0500 linux/amd64

~/go-review/src$ mote run gcc --version
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote363460504/bcmills-aix-ppc64-0.stdout"...
gcc (GCC) 8.4.0

What did you do?

Ran go list -export -json=BuildID runtime/cgo before and after clearing the Go build cache.

~/go-review/src/cmd/dist$ mote run ./go/bin/go list -export -json=BuildID runtime/cgo
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote3714306807/bcmills-aix-ppc64-0.stdout"...
{
        "BuildID": "ogJ-10b1BIf2OIMT5g5K/4jyhoIAusu99uwtt-kuy"
}
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote3714306807/bcmills-aix-ppc64-0.stdout".

~/go-review/src/cmd/dist$ mote run ./go/bin/go clean -cache
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote225234382/bcmills-aix-ppc64-0.stdout"...
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote225234382/bcmills-aix-ppc64-0.stdout".

~/go-review/src/cmd/dist$ mote run ./go/bin/go list -export -json=BuildID runtime/cgo
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote3775804530/bcmills-aix-ppc64-0.stdout"...
{
        "BuildID": "ogJ-10b1BIf2OIMT5g5K/7hDfzCuKuu_KQFpt3mXk"
}
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote3775804530/bcmills-aix-ppc64-0.stdout".

Note that the first part of the build ID (describing the inputs to the build) is identical; however, the second part (describing the output of the compiler) is not.

What did you expect to see?

Because the runtime/cgo package is within GOROOT, its build should always be reproducible (see #50183), and so each fresh rebuild of the package should result in the same BuildID, and any package in GOROOT that uses it (such as net) should also have a reproducible build.

Setting CGO_ENABLED=0 does result in a reproducible build for the net package on the aix-ppc64 builder:

~/go-review/src/cmd/dist$ gomote run -e 'CGO_ENABLED=0' bcmills-aix-ppc64-0 ./go/bin/go list -export -json=BuildID net
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote3253411634/bcmills-aix-ppc64-0.stdout"...
{
        "BuildID": "_EPUfZVuKfZHEmEBc02h/7QWWxSr9bui2XajXR10N"
}
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote3253411634/bcmills-aix-ppc64-0.stdout".

~/go-review/src/cmd/dist$ gomote run -e 'CGO_ENABLED=0' bcmills-aix-ppc64-0 ./go/bin/go clean -cache
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote3293010067/bcmills-aix-ppc64-0.stdout"...
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote3293010067/bcmills-aix-ppc64-0.stdout".

~/go-review/src/cmd/dist$ gomote run -e 'CGO_ENABLED=0' bcmills-aix-ppc64-0 ./go/bin/go list -export -json=BuildID net
# Streaming results from "bcmills-aix-ppc64-0" to "/tmp/gomote1809113884/bcmills-aix-ppc64-0.stdout"...
{
        "BuildID": "_EPUfZVuKfZHEmEBc02h/7QWWxSr9bui2XajXR10N"
}
# Wrote results from "bcmills-aix-ppc64-0" to "/tmp/gomote1809113884/bcmills-aix-ppc64-0.stdout".

And on linux/amd64 (and presumably other platforms), we do see reproducible builds for runtime/cgo even with cgo enabled:

~/go-review/src$ go list -export -json=BuildID runtime/cgo
{
        "BuildID": "WhJYk_P3Gk345k1UDp2X/iflidhKq7bnW0242XAPO"
}

~/go-review/src$ go clean -cache

~/go-review/src$ go list -export -json=BuildID runtime/cgo
{
        "BuildID": "WhJYk_P3Gk345k1UDp2X/iflidhKq7bnW0242XAPO"
}

What did you see instead?

With cgo enabled, the BuildID for the runtime/cgo and net packages on the aix-ppc64 builder differ each time the package is rebuilt from a clean cache.

(attn @golang/aix; CC @ianlancetaylor in case he has any tips on cgo reproducibility)

@bcmills bcmills added the OS-AIX label Nov 22, 2022
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 22, 2022
@gopherbot gopherbot added this to the Unreleased milestone Nov 22, 2022
@gopherbot
Copy link

Change https://go.dev/cl/452680 mentions this issue: cmd/dist: skip the staleness check for 'cmd' when testing on aix-ppc64

@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 22, 2022
gopherbot pushed a commit that referenced this issue Nov 22, 2022
The gcc toolchain on the aix-ppc64 builder apparently does not achieve
reproducible builds for packages that use cgo, which causes the
binaries in cmd that use package "net" (cmd/go, cmd/pprof, and
cmd/trace) to appear stale whenever the Go build cache is cleared.

For now, we work around the staleness by rebuilding std and simply not
checking whether cmd is stale.

For #56896.
Updates #47257.

Change-Id: I15f86e72dee53904b881710d5d5d613872361510
Reviewed-on: https://go-review.googlesource.com/c/go/+/452680
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Ayappan Perumal <ayappanec@gmail.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/452775 mentions this issue: cmd/dist: skip rebuilding packages during builder testing

@bcmills bcmills changed the title x/build: aix-ppc64 builder failing cmd/dist staleness checks at CL 452457 due to non-reproducible cgo builds x/build: aix-ppc64 builder has non-reproducible cgo builds Nov 22, 2022
@bcmills
Copy link
Member Author

bcmills commented Nov 22, 2022

The builder is once again passing (by skipping the staleness checks), so I've retitled the issue to reflect the underlying reproducibility problem that remains.

@laboger
Copy link
Contributor

laboger commented Nov 23, 2022

@ayappanec Would you look into this?

@ayappanec
Copy link

Yes, Looking into it

@rsc
Copy link
Contributor

rsc commented Dec 2, 2022

For what it's worth, CL 454836 is going to start building cmd without cgo, which will eliminate this as a concern even if it doesn't fix the underlying problem.

@bcmills
Copy link
Member Author

bcmills commented Dec 2, 2022

@cherrymui identified a source of nondeterminism in #57035. Does that explain this platform too?

(Does patching in https://go.dev/cl/454838 fix the symptoms reported here?)

@ayappanec
Copy link

GCC in AIX uses AIX native linker and assembler. I see the assembler always creates non-deterministic object files in AIX. So probably this looks like a limitation with AIX build

@laboger
Copy link
Contributor

laboger commented Dec 5, 2022

@ayappanec Can you build with https://go-review.googlesource.com/c/go/+/454838 and see if that resolves the nondeterminism?

@cherrymui
Copy link
Member

I tried. It doesn't...

gopherbot pushed a commit that referenced this issue Jan 30, 2023
Since packages in "std" no longer have install targets, checking them
for staleness is somewhat meaningless: if they are not cached they
will be rebuilt anyway, and without installed archives against which
we can compare them the staleness check will not detect builder skew.

It would still be meaningful to check "cmd" for staleness, but
(especially on sharded VM-based builders) that is a fairly expensive
operation relative to its benefit. If we are really interested in
detecting builder skew and/or build reproducibility, we could instead
add a "misc" test (similar to "misc/reboot", or perhaps even a part of
that test) that verifies that bootstrapped binaries are reproducible.

For #57734.
Updates #47257.
Updates #56896.

Change-Id: I8683ee81aefe8fb59cce9484453df9729bdc587c
Reviewed-on: https://go-review.googlesource.com/c/go/+/452775
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Austin Clements <austin@google.com>
johanbrandhorst pushed a commit to Pryz/go that referenced this issue Feb 12, 2023
Since packages in "std" no longer have install targets, checking them
for staleness is somewhat meaningless: if they are not cached they
will be rebuilt anyway, and without installed archives against which
we can compare them the staleness check will not detect builder skew.

It would still be meaningful to check "cmd" for staleness, but
(especially on sharded VM-based builders) that is a fairly expensive
operation relative to its benefit. If we are really interested in
detecting builder skew and/or build reproducibility, we could instead
add a "misc" test (similar to "misc/reboot", or perhaps even a part of
that test) that verifies that bootstrapped binaries are reproducible.

For golang#57734.
Updates golang#47257.
Updates golang#56896.

Change-Id: I8683ee81aefe8fb59cce9484453df9729bdc587c
Reviewed-on: https://go-review.googlesource.com/c/go/+/452775
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Austin Clements <austin@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-AIX
Projects
None yet
Development

No branches or pull requests

6 participants