WIP enhancements

[hannesm]

Hi Raja,

I worked a bit on orb in a slightly different direction: instead of building multiple times in parallel the same package (in different build paths), the build-path is fixed now, and a package is built twice in sequence. Sorry for the mess. Some things I added to orb: specify repositories via command-line, allow compiler git pins as base.

There are two subcommands: orb and rebuild. orb package is used to initially build a package, and creating a /tmp/bi-<name>-yyy-zzz/ directory populated with data to rebuild the exact same binary (using orb rebuild, or orb orb --twice (--diffoscope)):

• package.opam-switch -- a full opam export of the installed switch (requires Include extra-files in switch export if full is provided, on import create an overlay directory with the file contents ocaml/opam#4040 to embed files)
• package.build-environment.0 -- the set environment variables (atm SWITCH_PATH and SOURCE_DATE_EPOCH, as well as OS, OS_VERSION, OS_DISTRIBUTION, OS_FAMILY from opam)
• package.build-hashes.0 -- containing the hashes of the package (from dirtracking) I encountered DirTrack: check both size and mtime for cached entries ocaml/opam#4038 on the way, which is an actual bug.

This information (still slightly incomplete, thinking about host system packages and local pins to git branches / file system -- which imho should be improved in opam export) is sufficient to rebuild an opam package, which will hopefully lead to the same hash:

The subcommand rebuild, takes /tmp/bi-package-yyy-zzz as argument, reads package.opam-switch and finds a suitable build-environment (comparing the OS* variables above), and installs the package and outputs another package.build-environment.N and package.build-hashes.N. The hashes from the former installation are then compared to the new hashes. If the old build dir is still around and diffoscope is installed, it is used to compare all files that are different.

With the patches (using the orb branch of opam in https://github.com/hannesm/opam) below, I did some practical experiments on MirageOS unikernels -- which once configured are an opam package that installs a single ELF binary. A lot of them (using ~150 opam packages) are already reproducible! :)

This PR is work in progress, I hope to extend it a bit further before it is really ready to be used. Many thasnk for orb, without it I would not have come so far.

TODO (esp. the depext I have no good idea about and welcome any feedback)

• record os-family / os-distribution (is this sufficient? what about release versions? will opam-depext integration help here?)
• linux kernel version? freebsd system?
• c compiler, as, linker -- versions (or output of --version)
• for cpuid and mirage-entropy, record even more environment information (available cpu features)
• record host system packages (I've no idea whether opam-depext allows this)
• compare checksums in rebuild
• allow for a list of file extensions to not compare (cmt/cmti are not reproducible, but this doesn't matter for the final product)
• allow a recursive mode that compares all libraries
• enhance opam export to contain unique resources (i.e. no git branches - substitute with commit id / error if any package requires local filesystem references)

[hannesm]

I worked a bit more on this branch, and am in respect of functionality pretty fine, I updated the comment above to reflect reality).

[hannesm]

I'm closing this PR. Our fork has a different goal than the initial utility (and both are useful indeed). Maybe one day we should discuss how to move forward (and/or rename our fork and cut a release).