Reproducible builds

[ianopolous]

An important project like ipfs that is network facing should absolutely have reproducible builds.

I was hoping the fix would be easy (adding -buildid=) to the makefile as in golang/go#34186 but I couldn't get this to work. I'm hoping it is still easy for someone with better knowledge of the build system here.

The easy test is to simply try building from a different dir on the same system.

[aschmahmann]

@ianopolous is this about not using -trimpath in our make file, I think that should make things not path dependent? We do use -trimpath when building for dist.ipfs.io but not in the go-ipfs repo.

From what I recall this is because using -trimpath breaks local go mod replaces which means that people who try and make plugins that target local builds of go-ipfs will fail precisely because of the stripping out of all directory information.

I'm not against adding a separate reproducible build command to the makefile, but it's not really something I'd want to use to deploy the binaries to dist since that can/should be generic across the various Go binaries that we build.

Note: ipfs/distributions is undergoing some changes to make things easier to work with. I'm pro making an easier path to verifying distributions, as I think it still has some gross hacks in place from before the existence of -trimpath.

[warpfork]

Quick 2cents: the Reproducible Builds community defines the goal as "deterministic given the same source"... while sort of tip-toeing around exactly what the definition of "source" in that phrase encompasses. Approximately everyone agrees that letting timestamps on files on the build environment affect outcomes is bad and we shouldn't count those as "source", for example. But whether things like "the build directory" count as "source" (and thus are allowed to affect the result, while still considering the build overall reasonably deterministic and reproducible) receives varying opinion.

My opinion on this is that considering the build directory part of "source", and thus requiring it to be affixed in order to expect reproducible results, is acceptable (not ideal, but acceptable). The reason I hold this position is because amount of barrier to reproduction that's imposed by requiring a build environment with that directory is pretty minimal. (Especially compared to the scale of barriers typical for other things reproducible builds might wrangle with, like timestamps, unmanaged network dependencies, randomization, concurrency-induced randomization, or perniciously OS/driver-dependent things like "what order does the dirent listing syscall return results", etc. Those are all things reproducible builds are very interested in stamping out because they're vile to try to reproduce. Making a dir? Well, it's unfortunate you can't make it work at "./", yeah, but... Making a dir under a fixed path is walk in the park, by contrast to all these other potential problems.)

If we have other irreproducibility problems when the Go compiler version and the build directory are affixed... then I too would be concerned about whatever those issues are.

[ianopolous]

As far as I can tell, at the moment there is no way for me to reproduce a build from dist.ipfs.io. If I check out the same git commit and make build with the exact same version of go then the results are different. I don't know if this is just a path thing or not. That's just the easiest way for me to prove it changes. If it is just the path, then the path should be published along with the other build information here https://dist.ipfs.io/go-ipfs/v0.7.0/build-info

As Eric mentions, if getting a reproducible build is just a matter of building from a certain path I'm okay with that as long as the path is published. Ideally it would be path independent (though not at the cost of breaking plugin builds if that is the case!).

[ianopolous]

I've just double checked and it is being built with trimpath=/home/ian/go However the build dir on my system is /home/ian/go-ipfs
I assume it got the other one from my GOPATH.

[aschmahmann]

@ianopolous something that's likely very confusing with trimpath, is that there is a difference between using the asm + gc flags trimpath as in https://github.com/ipfs/go-ipfs/blob/8e6358a4fac40577950260d0c7a7a5d57f4e90a9/mk/golang.mk#L15 and building with go build -trimpath.

Building with just -trimpath is newer and is intended to give reproducible builds (although there could certainly be bugs here), however it comes at the cost of making local plugin development miserable since you can't both 1) use go mod replace directives and 2) have a build that strips out all path information. The asm + gc flags trimpath still leaves some path information in which is why local plugin building works.

---

I thought that https://github.com/ipfs/distributions did result in reproducible builds in the past and it does now use go build -trimpath. However, it has a bit of a wonky setup (partially stemming from the lack of -trimpath in the past) and is not particularly fun to use right now.

Over the course of the 0.8.0 release we're likely going to be making the process of using ipfs/distributions easier to work with and during the cleanup should be able to simplify the build process so others can reproduce more easily. There may be a few tricky bugs to work out but anecdotally Go plugins basically refuse to work if the binary the plugin is built against does not match the one it's deployed with so if plugins are working now we have builds that are either reproducible or quite close.