New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go, cmd/cgo: repeatable builds on OS X with external linker #9206

Closed
minux opened this Issue Dec 4, 2014 · 23 comments

Comments

Projects
None yet
10 participants
@minux
Member

minux commented Dec 4, 2014

Right now, on unix, building a pure Go program multiple times will generate
identical binaries.

However, once cgo is used, the binary will contain $WORK path in the DWARF
sections (I checked on Darwin/amd64 that if I strip the binaries, they are
again identical.)


Producing identical binaries for the same source is very valuable in a lot
circumstances.

For example, it will allow us to track down the exact revision of the code
used to produce the binary. (Note the runtime version is already embedded
into every Go binary.)

It will also help producing binary releases that could be trusted (see
https://gitian.org/ for a similar attempt for legacy C/C++ builds).

This time, we should add tests (for example, build two versions of misc/cgo/test
and see if they are identical.)

Current problems:
1. $WORK path leakage in cgo builds.
2. $WORK path leakage in -cover and tests.

For those two, I propose that if -work is not used, strip out the io/ioutil.TempDir()
part of the paths (work directory is not preserved, those paths won't be meaningful
anyway).


Please triage this issue, thanks.
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Dec 5, 2014

Comment 1:

Status changed to Accepted.

@minux minux added accepted labels Dec 5, 2014

@bradfitz bradfitz modified the milestone: Go1.5 Dec 16, 2014

@bradfitz bradfitz removed the release-go1.5 label Dec 16, 2014

@jonboulle jonboulle referenced this issue Feb 21, 2015

Open

reproducible builds #10

0 of 4 tasks complete

@rsc rsc removed accepted labels Apr 14, 2015

@rsc rsc modified the milestones: Go1.6Early, Go1.5 Jun 29, 2015

@rsc rsc modified the milestones: Unplanned, Go1.6Early Nov 4, 2015

@bradfitz

This comment has been minimized.

Member

bradfitz commented May 5, 2016

Sounds like this has been fixed. See #13247.

@bradfitz bradfitz closed this May 5, 2016

@rasky

This comment has been minimized.

Member

rasky commented May 5, 2016

FWIW, I just checked on tip on a medium size project on darwin, and there are still two small differences:

--- ndsemu.0.hex    2016-05-05 21:10:35.000000000 +0200
+++ ndsemu.1.hex    2016-05-05 21:10:25.000000000 +0200
@@ -469452,7 +469452,7 @@
 0073bee0  00 a0 f6 b1 02 00 00 00  00 00 00 00 00 00 00 00  |................|
 0073bef0  02 00 00 00 64 00 00 00  00 00 00 00 00 00 00 00  |....d...........|
 0073bf00  26 00 00 00 64 00 00 00  00 00 00 00 00 00 00 00  |&...d...........|
-0073bf10  29 00 00 00 66 03 01 00  80 9a 2b 57 00 00 00 00  |)...f.....+W....|
+0073bf10  29 00 00 00 66 03 01 00  60 9a 2b 57 00 00 00 00  |)...f...`.+W....|
 0073bf20  71 00 00 00 26 03 00 00  10 39 00 04 00 00 00 00  |q...&....9......|
 0073bf30  7e 00 00 00 26 03 00 00  10 39 00 04 00 00 00 00  |~...&....9......|
 0073bf40  89 00 00 00 26 03 00 00  50 39 00 04 00 00 00 00  |....&...P9......|
@@ -493095,7 +493095,7 @@
 00798490  6c 64 65 72 73 2f 6c 77  2f 6a 64 62 6b 37 70 5f  |lders/lw/jdbk7p_|
 007984a0  64 34 67 6a 36 71 70 79  64 63 7a 70 62 77 32 30  |d4gj6qpydczpbw20|
 007984b0  38 30 30 30 30 67 6e 2f  54 2f 67 6f 2d 6c 69 6e  |80000gn/T/go-lin|
-007984c0  6b 2d 32 39 39 31 33 36  32 35 34 2f 67 6f 2e 6f  |k-299136254/go.o|
+007984c0  6b 2d 32 38 30 38 31 36  33 38 30 2f 67 6f 2e 6f  |k-280816380/go.o|
 007984d0  00 72 75 6e 74 69 6d 65  2e 74 65 78 74 00 67 6f  |.runtime.text.go|
 007984e0  2e 62 75 69 6c 64 69 64  00 6d 61 69 6e 2e 4e 65  |.buildid.main.Ne|
 007984f0  77 48 77 42 61 63 6b 75  70 52 61 6d 00 6d 61 69  |wHwBackupRam.mai|

The first one I guess it's a timestamp, the second seems the path to a temporary directory created by/for go-link.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented May 5, 2016

Reopening. As far as I know the problem is Darwin-specific.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented May 5, 2016

CC @neild

@neild

This comment has been minimized.

Contributor

neild commented May 5, 2016

Sigh.

@rasky

This comment has been minimized.

Member

rasky commented May 5, 2016

I guess it will keep regressing until some tests are added to trybots. It looks like the typical feature that is impossible not to break without tests.

@neild

This comment has been minimized.

Contributor

neild commented May 5, 2016

There is a test, actually: TestCgoConsistentResults in src/cmd/go.

Works On My Machine(tm). Try running that test:

cd src/cmd/go
go test . -run=Consistent -v

Do you see -fdebug-prefix-map in the compiler invocation, or is the test skipped because the compiler does not support that flag?

If the test passes for you, can you give me steps to reproduce the inconsistency?

Thanks.

@rasky

This comment has been minimized.

Member

rasky commented May 5, 2016

The test passes, and -fdebug-prefix-map is being passed.

To reproduce what I see, it should be sufficient something like this:

brew install sdl2
git clone https://github.com/rasky/ndsemu $GOPATH/src/ndsemu
cd $GOPATH/src/ndsemu
go get
go build ; md5sum ./ndsemu
go build ; md5sum ./ndsemu
@rasky

This comment has been minimized.

Member

rasky commented May 9, 2016

@neild let me know if you reproduce and/or you need more info; i'd like to make sure that the bug is properly filed (though I don't personally need a quick resolution).

@rasky

This comment has been minimized.

Member

rasky commented Jun 5, 2016

It looks like the problem here is that the external linker adds a symbol to the symbol table, containing the full path of the go.o file in the temporary directory.

I think we might need something similar to this fix:

putelfsyment(putelfstr("go.go"), 0, 0, STB_LOCAL<<4|STT_FILE, SHN_ABS, 0)

but for macho instead of elf.

@bradfitz bradfitz modified the milestones: Go1.8, Unplanned Aug 24, 2016

@infinity0

This comment has been minimized.

infinity0 commented Aug 24, 2016

@ianlancetaylor @bradfitz Hi, by "varying the build path" I mean we are doing the first build in /build-1st and the second build in /build-2nd, and the resulting binaries contain differences.

Example diff and a list of things we vary.

I originally thought to open a new bug, because this bug seems to be about cgo and $WORK. We have also experienced this cgo issue elsewhere which we document here, but the issue I described with .gopclntab is different.

@infinity0

This comment has been minimized.

infinity0 commented Aug 24, 2016

I could retitle the other bug to something a bit more specific, if you prefer. Such as "golang embeds build path into .gopclntab, causing loss of reproducibility".

@bradfitz

This comment has been minimized.

Member

bradfitz commented Aug 24, 2016

Oh, I see. They are slightly different bugs. Okay, we can track them separately. I'll reopen #16860 and we can give it a better title.

@quentinmit quentinmit added the NeedsFix label Oct 6, 2016

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 21, 2016

@ianlancetaylor or @neild, didn't you figure out the magic flag

@rsc rsc changed the title from cmd/go, cmd/cgo: repeatable builds to cmd/go, cmd/cgo: repeatable builds on OS X with external linker Oct 21, 2016

@rsc rsc modified the milestones: Go1.9, Go1.8 Oct 21, 2016

@rsc

This comment has been minimized.

Contributor

rsc commented Jun 22, 2017

Confirmed this is still a problem:

$ go build -ldflags -linkmode=external -o gofmt1 cmd/gofmt
$ go build -ldflags -linkmode=external -o gofmt2 cmd/gofmt
$ cmp gofmt1 gofmt2
gofmt1 gofmt2 differ: char 3355697, line 14335
$ hexdump -C gofmt1 >gofmt1.hex
$ hexdump -C gofmt2 >gofmt2.hex
$ diff gofmt1.hex gofmt2.hex
208488c208488
< 00333430  d4 15 4c 59 00 00 00 00  58 03 00 00 26 03 00 00  |..LY....X...&...|
---
> 00333430  dc 15 4c 59 00 00 00 00  58 03 00 00 26 03 00 00  |..LY....X...&...|
218403,218404c218403,218404
< 00359fe0  2f 67 6f 2d 6c 69 6e 6b  2d 34 34 37 31 39 35 39  |/go-link-4471959|
< 00359ff0  38 34 2f 67 6f 2e 6f 00  72 75 6e 74 69 6d 65 2e  |84/go.o.runtime.|
---
> 00359fe0  2f 67 6f 2d 6c 69 6e 6b  2d 32 30 38 38 31 36 34  |/go-link-2088164|
> 00359ff0  38 36 2f 67 6f 2e 6f 00  72 75 6e 74 69 6d 65 2e  |86/go.o.runtime.|
$ 

I have no solution. I tried hacking cmd/link to pass a relative path (just "go.o") to the external linker and then running the linker in the temp directory (to make the relative path valid), hoping that the linker would then record the relative path instead of the absolute path seen in the diff. But that didn't help. (Also, in general I'm not sure of the implications of changing the directory where the linker runs, but it didn't help anyway.)

If someone knows how to tell clang on macOS not to record the absolute object paths used during a link, please let us know.

@rsc rsc modified the milestones: Go1.10, Go1.9 Jun 22, 2017

@purpleidea

This comment has been minimized.

purpleidea commented Jun 23, 2017

@rsc Here are a list of known issues with reproducible builds: https://tests.reproducible-builds.org/debian/index_issues.html perhaps this one has been studied and solved before for other packages.

Golang-1.7 is listed here: https://tests.reproducible-builds.org/debian/issues/unstable/captures_build_path_issue.html and discusses some of the issues.

HTH

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jun 23, 2017

@purpleidea Note that there are no known problems with reproducible builds with Go when using GCC, only when using clang.

@purpleidea

This comment has been minimized.

purpleidea commented Jun 23, 2017

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jun 23, 2017

@purpleidea Thanks. Assuming that is not on MacOS, please file a different issue with a reproduction case.

arthuredelstein pushed a commit to arthuredelstein/tor-browser-bundle that referenced this issue Jul 17, 2017

David Fifield Georg Koppen
Build go-webrtc and snowflake in the mac pluggable-transports descrip…
…tor.

I had to apply two tricks to get a reproducible snowflake-client.

The first is to use faketime to eliminate some timestamps. There were 11
variable timestamps in the file. Through experimentation, I found that
10 of them were dependent on the Go runtime (recompiling Go caused them
to change) and 1 was dependent on snowflake-client itself (recompiling
snowflake-client with the same runtime changed only that 1 timestamp).
The underlying issue has to do with clang 3.8.0 on Darwin embedding
timestamps, unsolved in the Go issue tracker as of 13 days ago.
golang/go#9206 (comment)

The second is a sed command to clobber embedded paths of the form
/tmp/go-buildXXXXXXXXX and /tmp/go-link-XXXXXXXXX. Their presence is
caused by some combination of Clang and Darwin, and there is as yet no
known workaround upstream.

boklm added a commit to boklm/tor-browser-build that referenced this issue Jul 19, 2017

Build go-webrtc and snowflake for mac.
tor-browser-bundle.git author: David Fifield <david@bamsoftware.com>
tor-browser-bundle.git commit: 26e0cd44f2886bfad1c3d30844ff7a21eb9d0478

Commit message from the tor-browser-bundle.git commit:

Build go-webrtc and snowflake in the mac pluggable-transports descriptor.

I had to apply two tricks to get a reproducible snowflake-client.

The first is to use faketime to eliminate some timestamps. There were 11
variable timestamps in the file. Through experimentation, I found that
10 of them were dependent on the Go runtime (recompiling Go caused them
to change) and 1 was dependent on snowflake-client itself (recompiling
snowflake-client with the same runtime changed only that 1 timestamp).
The underlying issue has to do with clang 3.8.0 on Darwin embedding
timestamps, unsolved in the Go issue tracker as of 13 days ago.
golang/go#9206 (comment)

The second is a sed command to clobber embedded paths of the form
/tmp/go-buildXXXXXXXXX and /tmp/go-link-XXXXXXXXX. Their presence is
caused by some combination of Clang and Darwin, and there is as yet no
known workaround upstream.
@rsc

This comment has been minimized.

Contributor

rsc commented Nov 2, 2017

This is fixed as of now. There were a few different steps and steps backward.

@rsc rsc closed this Nov 2, 2017

@purpleidea

This comment has been minimized.

purpleidea commented Nov 2, 2017

@rsc Great, thanks! What does it mean: "There were a few different steps and steps backward." ?

Cheers!

@rsc

This comment has been minimized.

Contributor

rsc commented Nov 7, 2017

It means there were multiple things to fix (steps forward) and maybe also a few new bugs introduced (steps backward) that also needed fixing.

@golang golang locked and limited conversation to collaborators Nov 7, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.