cmd/dist: check that builds are reproducible #58884
Comments
rsc
added
the
NeedsFix
|
Tested on windows/amd64, result is OK.
|
|
Thanks @qmuntal. The go1.21repro4.src.tar.gz came out different. Usually that means either (1) generated files got generated differently, or (2) extra files were included unexpectedly, either because they existed locally or because the filtering in distpack needs revision. Diffing the outputs of tar tzvf on both source archives should make it clear. Edit: There shouldn't be generated files in the source archive so it should be (2). |
|
I've tried again in a clean environment and it didn't have me any diff. The first time I ran
|
|
Thanks @qmuntal. What is golden.txt and solaris.txt in the fc output? The golden.txt looks like maybe the listing for go1.21repro4.src.tar.gz (the source distribution) while the solaris.txt looks like maybe the listing for go1.21repro4.solaris-amd64.tar.gz (the solaris binary distribution). I expect those would be different. |
That's the right file name mapping. Do you need any other test? |
|
@qmuntal, if you can reproduce the mismatch, then the next step would be to diff the content of your generated source archive (../pkg/distpack/go1.21repro4.src.tar.gz) against the original https://swtch.com/tmp/go1.21repro4.src.tar.gz that you started with. Something is different between those. Don't worry about binary archives like the solaris distribution until the source archive comes out right: if the source is different, the binaries will be different. The difference in the source is the root cause. Thanks! |
For #24904, we have made changes to Go 1.21 to make the Go distribution builds trivially reproducible: given the source archive and a Go bootstrap toolchain that is new enough (Go 1.21 requires Go 1.17 or later), running
make.bashormake.batshould produce the same binaries no matter the details of the host system, and adding the-distpackflag should produce the same archives no matter the details of the host system. Setting GOOS and GOARCH during this command cross-compiles a distribution for another system, and those should also be the same archives no matter the details of the host system. For example, building a linux/amd64 distribution should produce the same archive no matter whether the build happens on Linux, macOS, or Windows; no matter whether the host system is an x86 or an arm64; no matter where the archive is extracted; and so on.The source archive https://swtch.com/tmp/go1.21repro4.src.tar.gz holds the source tree for a Go release claming to be
go1.21repro4. The first 64 bits of its SHA256 hash are0322e4c62dd8d770(useopenssl sha256 go1.21repro4.src.tar.gzon Unix orcertutil -hashfile go1.21repro4.src.tar.gz sha256on Windows).If you expand that source archive and
cdintogo/srcand run./repro.bash(orrepro.baton Windows), it will run for quite a while building distribution archives for various systems. If you save the output and pipe it throughgrep distpack:, the output should match the canonical hashes here. For example:or on Windows:
As noted in the comments, do not write
repro.txtinto the current directory, or else it will be included in the archives and affect their hashes.You can test a single build using
Omit GOOS and GOARCH to test the build for the local system. Note that the canonical linux-arm build also sets GOARM=6. Other variables like CC and CGO_ENABLED should be unset.
If you find a system configuration where the script runs a build successfully but produces a different archive hash than the canonical ones, please check what is different by comparing against the reference archives. Good ways to identify differences include:
unzip -lvon each archive and diffing those outputstar tzvfon each archive and diffing those outputsdiff-rSometimes the difference will be in your environment configuration, such as setting CC or CGO_ENABLED causing changes in the defaults baked into the toolchain. Those kinds of differences due to configuration are expected. If you find a difference that's not caused by Go configuration, please file an issue with subject
cmd/distpack: reproducibility bug for GOOS/GOARCH(filling in GOOS and GOARCH) along with details of which files are different, and then mention the issue in a comment on this issue as well.I have already tested repro.bash on darwin/amd64, darwin/arm64, linux/amd64, and windows/arm64 systems, and I've tested every possible Go release from Go 1.17 onward as bootstrap toolchain when building on darwin/amd64. Of course, there may well still be bugs in Go setups I have not thought to test, and if so we want to find them. If there are any remaining, the first person to identify each new reproducibility bug root cause wins a gopher.
The text was updated successfully, but these errors were encountered: