FYI: comments on reproducible builds on Ubuntu 18.04 bionic and 18.10 cosmic

[riyad]

I've used diffoscope to compare two of my cosmic builds and @Eloston's bionic build against my own and I must say ungoogled-chromium seems reproducible.

Cosmic

The differences are:

• all deb files have the same content/hash ✔️
• all files have different file modification times. This is expected. As it's only metadata this doesn't change the file contents. ✔️
• foo.buildinfo records the build time. This is the only difference between both files' content. 😐
• which sadly foo.changes records as it saves the hash of every file. Hence this file's content differs in the recorded file hash of the foo.buildinfo file. 😞

cosmic.diffoscope.txt

Bionic

• all deb files have the same content/hash ✔️ although @Eloston's debs seem to be named slightly differently (.bionic vs ~bionic).
• all files have different file modification times and group permissions. This is expected (except for permissions). As it's only metadata this doesn't change the file contents. ✔️
• Since foo.changes records the hash of every file. This file's content differs in the recorded file hash of the foo.buildinfo file. 😞
• Although the produced deb files are the same the foo.buildinfo seems to have quite a few differences.
  • time of build
  • several differences in the Installed-Build-Depends: section. I'm not sure how those can differ. I just installed all packages listed by dpkg-checkbuilddeps. This still didn't seem to impact the resulting binaries though. o.O

bionic.diffoscope.txt

Builds

All builds were done in fresh LXD/LXC containers with the respective distro image. Bionic builds additionally used upstream LLVM. I can provide Ansible roles if anybody is interested.

[Eloston]

Thanks for testing and writing up your results. I am new to the subject of reproducible builds, so these results are interesting to me.

In regards to the differences, you mentioned in both bionic and cosmic how it is just the buildinfo and changes files that vary. These are just plain-text metadata files, so I don't think it matters if they are identical. However, I still think we should consult Debian's documentation on reproducible builds for their practices here.

That being said, what would you like us to do with this information? Do you think ungoogled-chromium should start applying ideas of reproducible builds?

[riyad]

Me neither, but the warning at the top of the ungoogled-chromium downloads page sparked my interest. Once I had the build process somewhat automated I just tried to poke around diffing two of my builds out of curiosity. I was surprised by the outcome to be honest. I was even more surprised when comparing my build against yours. 🙂

I had watched a few talks about the reproducible builds effort and was aware that there are a myriad of possibilities for differences especially in a project the size of chromium. And a little research shows that chromium itself isn't reproducible yet ... it seems because of build ids and some debug info. Whatever the source, it seems it's patched out of ungoogled-chromium. 😄

I'm still not sure what the consequence should be. We should be aware that builds on other platforms might not be reproducible ... although I'm fairly confident they are since AFAIK Windows and macOS builds also use the LLVM toolchain (instead of "platform specific" ones).

One idea would be to ask previous/new contributors of binaries to try to verify on their platform/distro.
Another would be to invert the assumption on the download page to e.g. "(preliminary) tests have shown builds should be/can be expected to be reproducible" and ask users for ideas and feedback.
Maybe gathering some more data and/or making this info public will spark some idea. 🤷‍♂️

I was just the lucky finder ... I don't know what to do with it. 😂