Repository navigation
Description
What version of Go are you using (go version)?
$ go version go1.14.1
Also tested with all versions from go1.13 and up, same problem.
Does this issue reproduce with the latest release?
Latest release, yes. Haven't tried tip.
What operating system and processor architecture are you using (go env)?
I ran into this when building the same code on many different machines: freebsd12 freebsd/386, debian10-stable linux/386, ubuntu18lts linux/amd64, debian-unstable linux/amd64, macos-latest darwin/amd64.
What did you do?
I built the same code on all the machines, with options that should result in the exact same binary, but got different results on same of the machines. I tracked it down to the machines with a single cpu creating a different binary. Building with GO19CONCURRENTCOMPILATION=0 on the multi-cpu machines resulted in binaries identical to the ones produced by the single-cpu machines.
Here is an example to run on a multi-cpu machine (I used linux/amd64, but should not make a difference):
git clone https://github.com/golang/tools cd tools git checkout a49f79bcc2246daebe8647db377475ffc1523d7b # latest at time of writing, no special meaning cd cmd/present # build GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=amd64 $HOME/sdk/go1.14.1/bin/go build -trimpath -a -ldflags=-buildid= sha256sum present # e14b34025c0e9bc8256f05a09d245449cfd382c2014806bd1d676e7b0794a89f present cp present present.morecpu # keep for diff, later on # build, now with GO19CONCURRENTCOMPILATION=0 GO19CONCURRENTCOMPILATION=0 GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=amd64 $HOME/sdk/go1.14.1/bin/go build -trimpath -a -ldflags=-buildid= sha256sum present # 44e21497225e8d88500b008ec961d64907ca496104a18802aaee817397c4fb11 present
What did you expect to see?
The exact same binary.
What did you see instead?
Different binaries.
Details
I found this problem by building on a single-cpu VM, where NumCPU is 1. That probably prevents concurrency during compiles. GO19CONCURRENTCOMPILATION=0 disables concurrent compiles. The compiler and linker use runtime.NumCPU() in a few places, and perhaps GO19CONCURRENTCOMPILATION=0 isn't enough but just hides the symptoms.
FYI, so far I always got the same binary on machines with multiple cpu's, whether 2 cpu's or more, like 8 cpu's. But perhaps that's just because a high level of parallelism isn't reached.
The problem does not manifest for very simple programs (e.g. goimports in the same git checkout). Perhaps because there isn't enough to parallelize.
When building with -ldflags=-w, omitting the DWARF symbol table, the two build commands produce the same output again. I looked into that because of the diff below. I've seen earlier "reproducible build commands" that included -ldflags="-s -w". I don't expect those to be required to get reproducible builds.
$ diff <(objdump -x present.morecpu) <(objdump -x present) 2,3c2,3 < present.morecpu: file format elf64-x86-64 < present.morecpu --- > present: file format elf64-x86-64 > present 60c60 < 18 .zdebug_loc 00338d92 0000000001049464 0000000001049464 00c19464 2**0 --- > 18 .zdebug_loc 00338d92 0000000001049457 0000000001049457 00c19457 2**0 62c62 < 19 .zdebug_ranges 0010d750 00000000010e2de1 00000000010e2de1 00cb2de1 2**0 --- > 19 .zdebug_ranges 0010d750 00000000010e2dd6 00000000010e2dd6 00cb2dd6 2**0
Activity
bcmills commentedon Mar 25, 2020
CC @randall77 @dr2chase @cherrymui @josharian
bcmills commentedon Mar 25, 2020
Tentatively milestoning as a Go 1.15 release blocker: @jayconrod has been working to iron out the remaining reproducibility bugs, and it would be really unfortunate to miss that target by this one issue.
(CC @matloob)
[-]cmd/go: builds not reproducible on single-cpu vs multiple-cpu machines[/-][+]cmd/compile: builds not reproducible on single-cpu vs multiple-cpu machines[/+]ianlancetaylor commentedon Mar 25, 2020
CC @thanm because DWARF.
thanm commentedon Mar 25, 2020
I'll take a look.
thanm commentedon Mar 25, 2020
I can reproduce the problem pretty easily. Issue is not with .debug_loc/.debug_range but with .debug_info, I am working on identifying a root cause.
thanm commentedon Mar 26, 2020
This is a pretty interesting bug to work on. Normally when debugging such problems "-S" and "-m" are your friends, but those are not an option when the concurrent back end is turned on.
One of the functions where I can see the problem in the cmd/present build is the method encoding/asn1.BitString.At. This function winds up with different DWARF depending on whether the parallel back end is on.
What's interesting about this method is that it has a value receiver, and the compiler decides at some point that it's going to generated a pointer receiver wrapper (in genwrapper), e.g. encoding/asn1.(*BitString).At.
When the parallel back end is not on, the sequence of events is:
When the parallel back end runs, the sequence of events is:
At first glance this doesn't look so bad, but it turns out that there is actually IR sharing going on between the two routines: their fn.Dcl lists point to the same nodes for params, variables, etc). This means that the actions of the inliner are going to wind up making perturbations that are going to be visible the DWARF gen code, which is bad.
I'm still thinking about the best way to fix this; I need to fig into the wrapper generation code to understand exactly why/how there is sharing and what the story is there.
thanm commentedon Apr 1, 2020
Oops, got the wrong github user. Trying again: @jeremyfaller
aclements commentedon Apr 16, 2020
Ping @thanm. Any progress on this?
9 remaining items
josharian commentedon May 15, 2020
Both those CLs look pretty reasonable to me. Having seen them, I’m weakly inclined towards the deeper fix.
randall77 commentedon May 15, 2020
I'd like to propose a 5th option. Not sure if this would actually fix the issue, but...
Before compilation, can we iterate through xtop, find all the wrappers that need to be generated, and generate them? Including making any inline decisions for them.
Then we're not generating new functions to compile while compiling, which has always been a thorny source of nondeterminism.
thanm commentedon May 15, 2020
@randall77 yes, I think doing that would also fix the problem. I can send a CL for that as well. I wasn't aware that we frown on adding functions during compilation... if that's one of our goals then I think it would definitely be better to go with your option.
randall77 commentedon May 15, 2020
I think we just try to keep as many complicated bits out of the parallel phase as possible. We've historically had trouble with symbols, errors, etc. showing up in the parallel phase and having to jump through hoops to serialize them deterministically. The less we do in the parallel phase, the better, at least from a code maintenance perspective (i.e. as long as it is a tiny fraction of compile time).
thanm commentedon May 15, 2020
OK. Just to be clear, the damage in this specific bug is being done before the parallel phase starts-- the bad behavior in question can also be triggered by "go tool compile -d compilelater" with the concurrent back end turned off.
thanm commentedon May 15, 2020
@randall77 I made an attempt at your suggestion, but it seems more complicated than I thought.
In particular, with this code in "compile()":
Note the "n.Name.Addrtaken()" in the if guarding the dtypesym call. This property is set by the type checker, but also in the "order" phase. If I hoist the code above out into a loop prior to the compilation loop (as you suggested) it's going to see a smaller set of addr-taken variables, due to the fact that "order" hasn't run yet.
With this in mind I think it might be better to stick to option 4.
randall77 commentedon May 15, 2020
I'm not sure what that code has to do with my suggestion. I'm not suggesting that that code be moved.
I'm not suggesting we compile the wrappers early, just generate them.
I guess that would require removing the
funccompilecall at the end ofgenwrapper, and replacing that with adding the function to the xtop list instead?thanm commentedon May 15, 2020
This is what I tried to do. The problem is that without running "order" you don't have an accurate picture of "address taken". Here is a CL with my attempt (the new functionality is guarded by an environment variable):
https://go-review.googlesource.com/c/go/+/234115
I think this is what you were suggesting, but maybe I am missing something? In particular see the "PROBLEM" comment in "funcgenwrappers".
gopherbot commentedon May 18, 2020
Change https://golang.org/cl/234417 mentions this issue:
cmd/compile: add test case for issue 38068thanm commentedon May 19, 2020
Discussed in the compiler/runtime meeting today. The plan is move forward with https://go-review.googlesource.com/c/go/+/234178 as opposed to the other candidate fixes.
cmd/compile: add test case for issue 38068