cmd/dist: check that builds are reproducible #58884
Comments
rsc
added
the
NeedsFix
|
Tested on windows/amd64, result is OK.
|
|
Thanks @qmuntal. The go1.21repro4.src.tar.gz came out different. Usually that means either (1) generated files got generated differently, or (2) extra files were included unexpectedly, either because they existed locally or because the filtering in distpack needs revision. Diffing the outputs of tar tzvf on both source archives should make it clear. Edit: There shouldn't be generated files in the source archive so it should be (2). |
|
I've tried again in a clean environment and it didn't have me any diff. The first time I ran
|
|
Thanks @qmuntal. What is golden.txt and solaris.txt in the fc output? The golden.txt looks like maybe the listing for go1.21repro4.src.tar.gz (the source distribution) while the solaris.txt looks like maybe the listing for go1.21repro4.solaris-amd64.tar.gz (the solaris binary distribution). I expect those would be different. |
That's the right file name mapping. Do you need any other test? |
|
@qmuntal, if you can reproduce the mismatch, then the next step would be to diff the content of your generated source archive (../pkg/distpack/go1.21repro4.src.tar.gz) against the original https://swtch.com/tmp/go1.21repro4.src.tar.gz that you started with. Something is different between those. Don't worry about binary archives like the solaris distribution until the source archive comes out right: if the source is different, the binaries will be different. The difference in the source is the root cause. Thanks! |
|
Change https://go.dev/cl/483695 mentions this issue: |
During bootstrapping, cmd/dist writes a file indicating which GOOS/GOARCH combinations are valid, and which support cgo-enabled builds. That information previously went into the go/build package, but today it fits in more naturally in the internal/platform package (which already has a number of functions indicating feature support for GOOS/GOARCH combinations). Moreover, as of CL 450739 the cmd/go logic for determining whether to use cgo is somewhat more nuanced than the go/build logic: cmd/go checks for the presence of a C compiler, whereas go/build does not (mostly because it determines its configuration at package-init time, and checking $PATH for a C compiler is somewhat expensive). To simplify this situation, this change: - consolidates the “cgo supported” check in internal/platform (alongside many other platform-support checks) instead of making it a one-off in go/build, - and updates a couple of tests to use testenv.HasCGO instead of build.Default.CgoEnabled to decide whether to test a cgo-specific behavior. For #58884. For #59500. Change-Id: I0bb2502dba4545a3d98c9e915727382ce536a0f3 Reviewed-on: https://go-review.googlesource.com/c/go/+/483695 Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com>
|
@qmuntal, did you have any luck tracking down the differences? There was one recent reproducibility fix in https://go.dev/cl/502755, but it's not clear to me that that would account for your observations. |
|
@rsc @bcmills sorry for late response, I forgot about this issue. As I said in my previous comment, I ran the |
|
Change https://go.dev/cl/513975 mentions this issue: |
|
Change https://go.dev/cl/513700 mentions this issue: |
|
Change https://go.dev/cl/514415 mentions this issue: |
This makes make.bash pass without any warnings for $ clang --version Ubuntu clang version 14.0.0-1ubuntu1.1 Target: x86_64-pc-linux-gnu Thread model: posix InstalledDir: /usr/bin $ gcc --version gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0 Copyright (C) 2021 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. $ The change in src/liblink/asm5.c silences a warning building on modern macOS, which doesn't actually produce a working toolchain anyway, but it still seems worth silencing the warning. These warnings surface now in the reproducible builds reports (for example https://gorebuild.storage.googleapis.com/gorebuild.html, click on "Bootstrap go1.4"). I'd rather not look at them anymore. For #58884. Change-Id: I689c862ad360ca23153438f9e143a1cb840730e4 Reviewed-on: https://go-review.googlesource.com/c/go/+/514415 TryBot-Bypass: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com>
This command rebuilds or verifies all the artifacts posted on go.dev/dl for the latest supported releases (the last patch of the last two major releases, plus the most recent release candidate if we're approaching a new release). It is meant to be run by the Go team to update a status page that can be linked from reproducible-builds.org, but it is also meant to be run by anyone who wants to "trust but verify" the status page themselves. For golang/go#57120. For golang/go#58884. Change-Id: I80a70275c1821a66b6219d24f29c2d11bfe464a8 Reviewed-on: https://go-review.googlesource.com/c/build/+/513975 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Russ Cox <rsc@golang.org>
|
Change https://go.dev/cl/515415 mentions this issue: |
Issue 61513 is resolved so this path can be turned on now. Confirmed to still pass now that go1.21rc4 is out. It was the first release built using improvements from CL 512437. For golang/go#57120. For golang/go#58884. For golang/go#61513. Change-Id: Ie39765f8c7ba514dea2bfccf7c8ef8acc5822a22 Reviewed-on: https://go-review.googlesource.com/c/build/+/515415 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Russ Cox <rsc@golang.org> Auto-Submit: Russ Cox <rsc@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
We now have a command to reproduce Go builds posted on go.dev/dl. Add a dashboard that people can check to see its results. We should be able to link to this page from https://reproducible-builds.org/citests/. For golang/go#57120. For golang/go#58884. Change-Id: I0bd1f9c26a9a003aa1f301125083195fdeb048b4 Reviewed-on: https://go-review.googlesource.com/c/website/+/513700 Reviewed-by: Heschi Kreinick <heschi@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
|
Change https://go.dev/cl/515455 mentions this issue: |
The "not modified" response code is 304, not 206. Oops. Use named constants to avoid similar mistakes in the future. Also update rebuild template to show more version information. For golang/go#57120. For golang/go#58884. Change-Id: I2c3ddf25cede0b5a853fa971226463a997f168c7 Reviewed-on: https://go-review.googlesource.com/c/website/+/515455 TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
Change https://go.dev/cl/556075 mentions this issue: |
The "missing from posted archive" case was checking the wrong variable and could never trigger. Fortunately, it's fairly harmless, as missing files would still be caught by gorebuild thanks to check hitting a nil pointer dereference trying to compare the missing file. Check the right variable to fix the panic, and print the intended text. For golang/go#57120. For golang/go#58884. Change-Id: I4560a9cc6c53bca37283c004826d728e175a1ff1 Reviewed-on: https://go-review.googlesource.com/c/build/+/556075 Reviewed-by: Michael Pratt <mpratt@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
|
It seems that the change in GOARM (#62475 (comment)) was never documented. |
|
Change https://go.dev/cl/639455 mentions this issue: |
|
@andig Thanks, sent a couple of CLs to document this. Odd that |
|
Change https://go.dev/cl/639276 mentions this issue: |
For #24904 For #58884 Change-Id: I0fc61f1a0bc6ebb22bdfd7a7b13ef36c68879384 Reviewed-on: https://go-review.googlesource.com/c/go/+/639276 Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Commit-Queue: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
This was changed in CL 470695 but was not mentioned in the release notes. For golang/go#24904 For golang/go#58884 Change-Id: Ib952ce43c65696b3a6091a378587cab1299f30e3 Reviewed-on: https://go-review.googlesource.com/c/website/+/639455 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Auto-Submit: Ian Lance Taylor <iant@golang.org> Auto-Submit: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
For #24904, we have made changes to Go 1.21 to make the Go distribution builds trivially reproducible: given the source archive and a Go bootstrap toolchain that is new enough (Go 1.21 requires Go 1.17 or later), running
make.bashormake.batshould produce the same binaries no matter the details of the host system, and adding the-distpackflag should produce the same archives no matter the details of the host system. Setting GOOS and GOARCH during this command cross-compiles a distribution for another system, and those should also be the same archives no matter the details of the host system. For example, building a linux/amd64 distribution should produce the same archive no matter whether the build happens on Linux, macOS, or Windows; no matter whether the host system is an x86 or an arm64; no matter where the archive is extracted; and so on.The source archive https://swtch.com/tmp/go1.21repro4.src.tar.gz holds the source tree for a Go release claming to be
go1.21repro4. The first 64 bits of its SHA256 hash are0322e4c62dd8d770(useopenssl sha256 go1.21repro4.src.tar.gzon Unix orcertutil -hashfile go1.21repro4.src.tar.gz sha256on Windows).If you expand that source archive and
cdintogo/srcand run./repro.bash(orrepro.baton Windows), it will run for quite a while building distribution archives for various systems. If you save the output and pipe it throughgrep distpack:, the output should match the canonical hashes here. For example:or on Windows:
As noted in the comments, do not write
repro.txtinto the current directory, or else it will be included in the archives and affect their hashes.You can test a single build using
Omit GOOS and GOARCH to test the build for the local system. Note that the canonical linux-arm build also sets GOARM=6. Other variables like CC and CGO_ENABLED should be unset.
If you find a system configuration where the script runs a build successfully but produces a different archive hash than the canonical ones, please check what is different by comparing against the reference archives. Good ways to identify differences include:
unzip -lvon each archive and diffing those outputstar tzvfon each archive and diffing those outputsdiff-rSometimes the difference will be in your environment configuration, such as setting CC or CGO_ENABLED causing changes in the defaults baked into the toolchain. Those kinds of differences due to configuration are expected. If you find a difference that's not caused by Go configuration, please file an issue with subject
cmd/distpack: reproducibility bug for GOOS/GOARCH(filling in GOOS and GOARCH) along with details of which files are different, and then mention the issue in a comment on this issue as well.I have already tested repro.bash on darwin/amd64, darwin/arm64, linux/amd64, and windows/arm64 systems, and I've tested every possible Go release from Go 1.17 onward as bootstrap toolchain when building on darwin/amd64. Of course, there may well still be bugs in Go setups I have not thought to test, and if so we want to find them. If there are any remaining, the first person to identify each new reproducibility bug root cause wins a gopher.
The text was updated successfully, but these errors were encountered: