Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
cmd/go: support reproducible builds regardless of build path? #16860
It would be good if go was able to generate bit-for-bit identical binaries, even if the build environment changes in unimportant ways. (For example, so users can reproduce binaries without requiring root.)
Up until recently, we have been able to reproduce go binaries whilst modifying many parts of the build environment, except that we kept the build-path constant. However recently, we started to also vary the build-path. (Why "recently", is irrelevant to this report, but I can go into it if you ask.)
Anyway, now we can see that
In particular, I'm not sure how this will interfere with readers of this information. I know that
Also, the part where I comment out
The call to
However if you give me some guidelines on how to make this patch acceptable, I'll be happy to do this work and submit a PR.
Yeah, let's merge conversation in #9206.
@infinity0, instructions for sending a change are at https://golang.org/doc/contribute.html#Code_review which looks like a wall of text, but it's not many actual steps. Be sure to check errors, not have commented-out code, and include tests if possible.
referenced this issue
Aug 24, 2016
Thanks for the explanation (over on #9206). It's not clear to me that this should be a goal. I agree that reproducible builds are essential. However, it's not clear to me that reproducible builds when the sources are in different directories are essential.
For example, exactly the same thing happens when using GCC with the -g option. The source directory is included in the debug info, in the
Not including the source directory will make debugging more difficult.
For GCC, we are setting
Yes, we're being quite strict with ourselves in trying to make things independent of the build path, but we think it's a good goal that would allow more people in practice to perform build verification.
Ideally, we'd want as many people to rebuild the same hash as possible, so that the rest of the world (who we assume don't want to do these rebuilds) can see that 20 people built the same hash, rather than 5 people built 4 different hashes. It is indeed unclear at the moment what the "best tradeoff" is - we're just scratching the surface of this topic ourselves - but this particular issue seemed fairly easy to me to fix.
(edit: more accurate to say
Yes, "making debugging more difficult" was also my concern. FWIW, our experiments with GCC did not make things harder to debug (this was by chance, we had to try it out to see it). I can appreciate that Go is different, but if you can explain the details I could also try to think of solutions.
I think this is important. Debugging is also very important. From what I understand this is the issue of two users with different goroots set. Renaming of a module/forking it to a new repo name would not be supported. So, is the question how to share debug information? How to relate debug information created in one root to another root? If you are in the same root as the debug info was created it would be nice if we could strip out all the stuff outside the root.
The compiler has a
There are at least two other ways that directory names end up in binaries, even with cgo disabled. First,
These settings combined can get very close to bit-for-bit reproducible builds. The Go toolchain's build id also needs to be constrained, maybe via
To be clear, I don't recommend using these knobs ... unless their value is worth their complexity cost for your application, and you're prepared to own the pieces when it stops working or changes behavior on some future release.
Thanks for the detailed reply, @rhysh.
We started specifying the
I also prepared a patch which makes
I’ll send it for review once the merge window opens after the Go 1.9 release. This will at least give us some code to further inform the discussion.