Reproducible

[hannesm]

this is a draft to enable reproducible builds with mirageos 4.x

[hannesm]

The idea behind this PR is to embed all the required information into the opam file, so that -- similar to MirageOS 3 -- we can use the opam file (with a non-standard opam client which does the opam monorepo pull etc.) as build instructions for a unikernel.

Namely, what is embedded now:

• x-mirage-pre-build, a list of commands (similar to build) that specify mirage configure and make depend -- if executed outside of the opam lock, they'll succeed and prepare the build dir for executing the build step
• x-mirage-extra-repo, a list of name and url of repositories used during the make depend. I'm aware that the mirage tool has a hardcoded list of some defaults, but (a) that may change (b) users can specify on command line more overlay repositories.

The other change is that opam-monorepo is called as opam-monorepo instead of opam monorepo -- is this fine for you? The reason behind that is that we use a different opam client (https://github.com/roburio/orb) for conducting reproducible builds, thus there's no opam around (apart from a simple shell script that evaluates opam var / opam config var).

Do these changes look good for being merged? Another change in my mind is to re-add the cd to the build instrcutrions (the first return value from find_git in functoria/opam.ml) -- unclear to me why this was removed in the mirage3 -> mirage4 process. Is anyone using the build instructions of the generated opam files at all?

[dinosaure]

I look into the PR in details (specially about the extra-repo variable added into #1270) and it seems good for me. The CI is green, so if it helps you to provide a reproducible for MirageOS 4, I will happy to merge it. May be a double-check from @TheLortex.

[hannesm]

I'll comment here again once satisfied with the flow of build and rebuild. There may be some more changes needed for this.

[TheLortex]

Thank for this work, this is very nice ! I will take some time this week to review the PR.

[hannesm]

With the current version, I'm satisfied that a non-interactive non-opam driver can build unikernels using the mirage-generated opam file as follows:

• execute an opam install --deps-only .
• use the value of x-mirage-extra-repo to be added to the repositories
• execute the x-mirage-configure step
• execute x-mirage-pre-build (which execute make lock pull)
• gather extra duniverse sources from the x-mirage-opam-lock-location opam file (x-opam-monorepo-duniverse-dirs (this is needed for reproducing the exact same binary and keeping all urls to tarballs with checksums into a single place)) (of course the opam monorepo pull does this step automatically, the fetching)
• execute build and install instructions

So, with this PR, there are several fields added to the generated opam files:

• build only includes the mirage build term (but may include a cd <subdir> first, e.g. when building tutorial/hello from mirage-skeleton);
• install the same (includes the subdirectory)
• x-mirage-opam-lock-location a string with the path to the lockfile (tutorial/hello/mirage/hello-hvt.opam.locked)
• x-mirage-configure the configure that was executed (mirage configure -t hvt --no-extra-repo)
• x-mirage-pre-build the monorepo lock pull command

Wiht this PR, I'm able to conduct unikernel builds with orb -- which is great. I'm not yet able to conduct a rebuild (of the same binary), which should avoid the opam repository (repositories) and only use the tarballs from the opam switch export (which includes the x-opam-monorepo-duniverse-dirs for this purpose).

A review of this PR would be great, I suspect that the mirage-generated opam file is not used anywhere, so these changes should be fine.

[hannesm]

An example opam file of the tutorial/hello unikernel is located at https://git.robur.io/hannes/test-opam/src/branch/main/packages/mirage-unikernel-hello-hvt.0.0.1/opam

[hannesm]

TheLortex commented on 21 Jun

Thank for this work, this is very nice ! I will take some time this week to review the PR.

Any chance you had time to look into this? Would be great to have this reviewed and merged for the next mirage release.

[TheLortex]

The patch looks good, and indeed won't break anything as currently nothing makes use of the generated opam file. I have added two small comments.

Additionally, I think we should either (or both):

• document how to use the opam file with these extra fields. Maybe in opam.mli
• provide a minimal driver that is able to build an unikernel from the opam definition. This would enable having a command such as mirage-install https://git.robur.io/hannes/test-opam/raw/branch/main/packages/mirage-unikernel-hello-hvt.0.0.1/opam or mirage install <OPAM FILE PATH> depending on if we want that feature in the mirage tool or as an additional binary.

What do you think ?

[hannesm]

Thanks for your review and comments. I addressed your minor comments in subsequent commits.

document how to use the opam file with these extra fields. Maybe in opam.mli

Is such a documentation in opam.mli worth it? I'm more into writing a post for our website that describes these extra fields and how they can be used.

provide a minimal driver that is able to build an unikernel from the opam definition. This would enable having a command such as mirage-install https://git.robur.io/hannes/test-opam/raw/branch/main/packages/mirage-unikernel-hello-hvt.0.0.1/opam or mirage install depending on if we want that feature in the mirage tool or as an additional binary.

I've no clue what a "minimal driver" is. From my perspective, orb (the opam reproducible builder) is the minimal application that takes such an mirage-generated opam file as input and outputs a unikernel binary (and sufficient information to reproduce the exact same binary). If you like to develop something even smaller (mirage install?), please give it a try. For me, orb is mostly unrelated and independent of mirage -- thus its dependency cone does neither include "mirage" nor "opam" being available on the host system (and thus can be used in minimal systems (jail/container).

[TheLortex]

Is such a documentation in opam.mli worth it? I'm more into writing a post for our website that describes these extra fields and how they can be used.

A blog post would be nice indeed. It's just that the link between these fields and orb or reproducible builds is not clear for the unsuspecting user that woud look into these generated files, definitely that's not critical though.

I've no clue what a "minimal driver" is. From my perspective, orb (the opam reproducible builder) is the minimal application that takes such an mirage-generated opam file [...]

I've looked into your MirageOS 4 work for orb : I meant something like that, so indeed there's no point duplicating with something else. Instead we can add a Reproducible builds paragraph in the README to explain the workflow and add a pointer to orb. It can be done in another PR.

[hannesm]

Thanks @TheLortex, I'll merge and cut a release soon.