Skip to content

cmd/compile: builds not reproducible on single-cpu vs multiple-cpu machines #38068

Closed

Description

@mjl-

What version of Go are you using (go version)?

$ go version
go1.14.1

Also tested with all versions from go1.13 and up, same problem.

Does this issue reproduce with the latest release?

Latest release, yes. Haven't tried tip.

What operating system and processor architecture are you using (go env)?

I ran into this when building the same code on many different machines: freebsd12 freebsd/386, debian10-stable linux/386, ubuntu18lts linux/amd64, debian-unstable linux/amd64, macos-latest darwin/amd64.

What did you do?

I built the same code on all the machines, with options that should result in the exact same binary, but got different results on same of the machines. I tracked it down to the machines with a single cpu creating a different binary. Building with GO19CONCURRENTCOMPILATION=0 on the multi-cpu machines resulted in binaries identical to the ones produced by the single-cpu machines.

Here is an example to run on a multi-cpu machine (I used linux/amd64, but should not make a difference):

git clone https://github.com/golang/tools
cd tools
git checkout a49f79bcc2246daebe8647db377475ffc1523d7b  # latest at time of writing, no special meaning
cd cmd/present

# build
GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=amd64 $HOME/sdk/go1.14.1/bin/go build -trimpath -a -ldflags=-buildid=

sha256sum present
# e14b34025c0e9bc8256f05a09d245449cfd382c2014806bd1d676e7b0794a89f  present

cp present present.morecpu  # keep for diff, later on

# build, now with GO19CONCURRENTCOMPILATION=0
GO19CONCURRENTCOMPILATION=0 GO111MODULE=on CGO_ENABLED=0 GOOS=linux GOARCH=amd64 $HOME/sdk/go1.14.1/bin/go build -trimpath -a -ldflags=-buildid=

sha256sum present
# 44e21497225e8d88500b008ec961d64907ca496104a18802aaee817397c4fb11  present

What did you expect to see?

The exact same binary.

What did you see instead?

Different binaries.

Details

I found this problem by building on a single-cpu VM, where NumCPU is 1. That probably prevents concurrency during compiles. GO19CONCURRENTCOMPILATION=0 disables concurrent compiles. The compiler and linker use runtime.NumCPU() in a few places, and perhaps GO19CONCURRENTCOMPILATION=0 isn't enough but just hides the symptoms.

FYI, so far I always got the same binary on machines with multiple cpu's, whether 2 cpu's or more, like 8 cpu's. But perhaps that's just because a high level of parallelism isn't reached.

The problem does not manifest for very simple programs (e.g. goimports in the same git checkout). Perhaps because there isn't enough to parallelize.

When building with -ldflags=-w, omitting the DWARF symbol table, the two build commands produce the same output again. I looked into that because of the diff below. I've seen earlier "reproducible build commands" that included -ldflags="-s -w". I don't expect those to be required to get reproducible builds.

$ diff <(objdump -x present.morecpu) <(objdump -x present)
2,3c2,3
< present.morecpu:     file format elf64-x86-64
< present.morecpu
---
> present:     file format elf64-x86-64
> present
60c60
<  18 .zdebug_loc   00338d92  0000000001049464  0000000001049464  00c19464  2**0
---
>  18 .zdebug_loc   00338d92  0000000001049457  0000000001049457  00c19457  2**0
62c62
<  19 .zdebug_ranges 0010d750  00000000010e2de1  00000000010e2de1  00cb2de1  2**0
---
>  19 .zdebug_ranges 0010d750  00000000010e2dd6  00000000010e2dd6  00cb2dd6  2**0

Activity

  1. bcmills commented on Mar 25, 2020

    @bcmills
    Contributor
  2. added
    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
    on Mar 25, 2020
  3. added this to the Go1.15 milestone on Mar 25, 2020
  4. bcmills commented on Mar 25, 2020

    @bcmills
    Contributor

    Tentatively milestoning as a Go 1.15 release blocker: @jayconrod has been working to iron out the remaining reproducibility bugs, and it would be really unfortunate to miss that target by this one issue.

    (CC @matloob)

  5. changed the title [-]cmd/go: builds not reproducible on single-cpu vs multiple-cpu machines[/-] [+]cmd/compile: builds not reproducible on single-cpu vs multiple-cpu machines[/+] on Mar 25, 2020
  6. ianlancetaylor commented on Mar 25, 2020

    @ianlancetaylor
    Contributor

    CC @thanm because DWARF.

  7. thanm commented on Mar 25, 2020

    @thanm
    Contributor

    I'll take a look.

  8. self-assigned this
    on Mar 25, 2020
  9. thanm commented on Mar 25, 2020

    @thanm
    Contributor

    I can reproduce the problem pretty easily. Issue is not with .debug_loc/.debug_range but with .debug_info, I am working on identifying a root cause.

  10. thanm commented on Mar 26, 2020

    @thanm
    Contributor

    This is a pretty interesting bug to work on. Normally when debugging such problems "-S" and "-m" are your friends, but those are not an option when the concurrent back end is turned on.

    One of the functions where I can see the problem in the cmd/present build is the method encoding/asn1.BitString.At. This function winds up with different DWARF depending on whether the parallel back end is on.

    What's interesting about this method is that it has a value receiver, and the compiler decides at some point that it's going to generated a pointer receiver wrapper (in genwrapper), e.g. encoding/asn1.(*BitString).At.

    When the parallel back end is not on, the sequence of events is:

    == ARGS: -c=1
    | compile "".BitString.At
    | dwarfGen for "".BitString.At
    | genwrapper creates "".(*BitString).At
    | inline "".BitString.At into "".(*BitString).At
    | compile "".(*BitString).At
    | dwarfGen for "".(*BitString).At
    

    When the parallel back end runs, the sequence of events is:

    == ARGS: -c=8
    genwrapper creates "".(*BitString).At
    inline "".BitString.At into "".(*BitString).At
    <in parallel>
    | compile "".BitString.At         |    compile "".(*BitString).At
    | dwarfGen for "".BitString.At    |    dwarfGen for "".(*BitString).At
    

    At first glance this doesn't look so bad, but it turns out that there is actually IR sharing going on between the two routines: their fn.Dcl lists point to the same nodes for params, variables, etc). This means that the actions of the inliner are going to wind up making perturbations that are going to be visible the DWARF gen code, which is bad.

    I'm still thinking about the best way to fix this; I need to fig into the wrapper generation code to understand exactly why/how there is sharing and what the story is there.

  11. added
    NeedsFixThe path to resolution is known, but the work has not been done.
    and removed
    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
    on Mar 31, 2020
  12. thanm commented on Apr 1, 2020

    @thanm
    Contributor

    Oops, got the wrong github user. Trying again: @jeremyfaller

  13. aclements commented on Apr 16, 2020

    @aclements
    Member

    Ping @thanm. Any progress on this?

  14. 9 remaining items

  15. josharian commented on May 15, 2020

    @josharian
    Contributor

    Both those CLs look pretty reasonable to me. Having seen them, I’m weakly inclined towards the deeper fix.

  16. randall77 commented on May 15, 2020

    @randall77
    Contributor

    I'd like to propose a 5th option. Not sure if this would actually fix the issue, but...

    Before compilation, can we iterate through xtop, find all the wrappers that need to be generated, and generate them? Including making any inline decisions for them.
    Then we're not generating new functions to compile while compiling, which has always been a thorny source of nondeterminism.

  17. thanm commented on May 15, 2020

    @thanm
    Contributor

    @randall77 yes, I think doing that would also fix the problem. I can send a CL for that as well. I wasn't aware that we frown on adding functions during compilation... if that's one of our goals then I think it would definitely be better to go with your option.

  18. randall77 commented on May 15, 2020

    @randall77
    Contributor

    I think we just try to keep as many complicated bits out of the parallel phase as possible. We've historically had trouble with symbols, errors, etc. showing up in the parallel phase and having to jump through hoops to serialize them deterministically. The less we do in the parallel phase, the better, at least from a code maintenance perspective (i.e. as long as it is a tiny fraction of compile time).

  19. thanm commented on May 15, 2020

    @thanm
    Contributor

    OK. Just to be clear, the damage in this specific bug is being done before the parallel phase starts-- the bad behavior in question can also be triggered by "go tool compile -d compilelater" with the concurrent back end turned off.

  20. thanm commented on May 15, 2020

    @thanm
    Contributor

    @randall77 I made an attempt at your suggestion, but it seems more complicated than I thought.

    In particular, with this code in "compile()":

    	// Make sure type syms are declared for all types that might
    	// be types of stack objects. We need to do this here
    	// because symbols must be allocated before the parallel
    	// phase of the compiler.
    	for _, n := range fn.Func.Dcl {
    		switch n.Class() {
    		case PPARAM, PPARAMOUT, PAUTO:
    			if livenessShouldTrack(n) && n.Name.Addrtaken() {
    				dtypesym(n.Type)
    				// Also make sure we allocate a linker symbol
    				// for the stack object data, for the same reason.
    				if fn.Func.lsym.Func.StackObjects == nil {
    					fn.Func.lsym.Func.StackObjects = Ctxt.Lookup(fn.Func.lsym.Name + ".stkobj")
    				}
    			}
    		}
    	}
    

    Note the "n.Name.Addrtaken()" in the if guarding the dtypesym call. This property is set by the type checker, but also in the "order" phase. If I hoist the code above out into a loop prior to the compilation loop (as you suggested) it's going to see a smaller set of addr-taken variables, due to the fact that "order" hasn't run yet.

    With this in mind I think it might be better to stick to option 4.

  21. randall77 commented on May 15, 2020

    @randall77
    Contributor

    I'm not sure what that code has to do with my suggestion. I'm not suggesting that that code be moved.

    I'm not suggesting we compile the wrappers early, just generate them.
    I guess that would require removing the funccompile call at the end of genwrapper, and replacing that with adding the function to the xtop list instead?

  22. thanm commented on May 15, 2020

    @thanm
    Contributor

    not suggesting we compile the wrappers early, just generate them.

    This is what I tried to do. The problem is that without running "order" you don't have an accurate picture of "address taken". Here is a CL with my attempt (the new functionality is guarded by an environment variable):

    https://go-review.googlesource.com/c/go/+/234115

    I think this is what you were suggesting, but maybe I am missing something? In particular see the "PROBLEM" comment in "funcgenwrappers".

  23. gopherbot commented on May 18, 2020

    @gopherbot
    Contributor

    Change https://golang.org/cl/234417 mentions this issue: cmd/compile: add test case for issue 38068

  24. thanm commented on May 19, 2020

    @thanm
    Contributor

    Discussed in the compiler/runtime meeting today. The plan is move forward with https://go-review.googlesource.com/c/go/+/234178 as opposed to the other candidate fixes.

  25. locked and limited conversation to collaborators on May 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.okay-after-beta1Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1release-blocker

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions