Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP enhancements #1

wants to merge 54 commits into from


Copy link

@hannesm hannesm commented Dec 9, 2019

Hi Raja,

I worked a bit on orb in a slightly different direction: instead of building multiple times in parallel the same package (in different build paths), the build-path is fixed now, and a package is built twice in sequence. Sorry for the mess. Some things I added to orb: specify repositories via command-line, allow compiler git pins as base.

There are two subcommands: orb and rebuild. orb package is used to initially build a package, and creating a /tmp/bi-<name>-yyy-zzz/ directory populated with data to rebuild the exact same binary (using orb rebuild, or orb orb --twice (--diffoscope)):

This information (still slightly incomplete, thinking about host system packages and local pins to git branches / file system -- which imho should be improved in opam export) is sufficient to rebuild an opam package, which will hopefully lead to the same hash:

The subcommand rebuild, takes /tmp/bi-package-yyy-zzz as argument, reads package.opam-switch and finds a suitable build-environment (comparing the OS* variables above), and installs the package and outputs another and The hashes from the former installation are then compared to the new hashes. If the old build dir is still around and diffoscope is installed, it is used to compare all files that are different.

With the patches (using the orb branch of opam in below, I did some practical experiments on MirageOS unikernels -- which once configured are an opam package that installs a single ELF binary. A lot of them (using ~150 opam packages) are already reproducible! :)

This PR is work in progress, I hope to extend it a bit further before it is really ready to be used. Many thasnk for orb, without it I would not have come so far.

TODO (esp. the depext I have no good idea about and welcome any feedback)

  • record os-family / os-distribution (is this sufficient? what about release versions? will opam-depext integration help here?)
  • linux kernel version? freebsd system?
  • c compiler, as, linker -- versions (or output of --version)
  • for cpuid and mirage-entropy, record even more environment information (available cpu features)
  • record host system packages (I've no idea whether opam-depext allows this)
  • compare checksums in rebuild
  • allow for a list of file extensions to not compare (cmt/cmti are not reproducible, but this doesn't matter for the final product)
  • allow a recursive mode that compares all libraries
  • enhance opam export to contain unique resources (i.e. no git branches - substitute with commit id / error if any package requires local filesystem references)

Copy link

hannesm commented Dec 11, 2019

I worked a bit more on this branch, and am in respect of functionality pretty fine, I updated the comment above to reflect reality).

Copy link

hannesm commented Sep 26, 2022

I'm closing this PR. Our fork has a different goal than the initial utility (and both are useful indeed). Maybe one day we should discuss how to move forward (and/or rename our fork and cut a release).

@hannesm hannesm closed this Sep 26, 2022
@hannesm hannesm deleted the build-path-prefix-map branch September 26, 2022 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

Successfully merging this pull request may close these issues.

None yet

1 participant