Use Dune instead of OCamlbuild to build unikernels #969
I used several unikernel samples to explore what are the problems blocking the Dune transition. I've taken them from
Used features of OCamlbuild that aren't implemented in Dune:
Workaround to each problem
Packages to update
Issues and comments
How do you feel about this ? How should we proceed to update packages ? Am I missing something ? Feel free to comment !
The text was updated successfully, but these errors were encountered:
This plan sounds good to me!
About specific libraries:
After struggling with bigarray today I finally opted in for a hack: even if ocamlopt links
For mirage/mirage#969 Instead of pkg-config, one can use the following files to get the compilation flags: ocaml-freestanding/libs ocaml-freestanding/cflags
For mirage/mirage#969 Instead of PKG-CONFIG, one can use the following files to get compilation flags: mirage-xen-ocaml/libs mirage-xen-ocaml/cflags mirage-xen-posix/minios-cflags mirage-xen-posix/minios-libs mirage-xen-posix/posix-cflags mirage-xen-posix/posix-libs
New update for
As discussed before, I feel like this should be a first class feature in dune. Making existing libraries virtual is something that is always going to come up if this feature gains any traction. Dune should properly support such transitions instead of having the users pollute the package namespace.
I'm trying to argue that this is making a bad situation worse.
Because even right now, Mirage is a monorepo project.
In fact, here is something missing from https://mirage.io:
MirageOS is not, in general, compatible with third-party OCaml code. Users cannot expect separately developed OCaml libraries to work in the unikernel environment, unless and until they are accepted into the MirageOS project and ported accordingly.
And while this could be phrased a little less directly, it cannot be made more true. It's not even by design; this is just a consequence of long-standing technical debt.
Just to put everyone on the same page, please bear with me for a moment, while I try to give some context:
Up until 2015 this was quite obvious. For example, there was a copy of
As more C came into play, it progressed to the next stage -- in order to run in a unikernel, every library had to build its stubs for all mirage targets and broadcast its intentions to the
This gave us several
From there, we've made the first steps towards making this all-in-one approach a formal reality.
Which brings us to the present. The current proposal is a perfectly logical consequence of bringing everything that could possibly be compiled into a unikernel under one roof (directly, by forking, or through an overlay). With everything building from a single project, there is nothing odd about a compilation scheme which requires
because what is being proposed here is understood as a Mirage-internal refactoring. As far as I can tell, the current state of Mirage is to completely embrace this idea that separate development is off the table.
If everything is not understood as simply being part of Mirage, then this scheme is so prohibitively invasive to the point of being anti-modular.
Now, I really am glad that we can agree on the ultimate vision, but we can probably all agree on what we want to have at the point at infinity. Namely, everything.
What is more interesting is whether the next step will
If it increases, it builds a larger wall around Mirage.
And if history is any guideline, people stop working on a problem the very moment any solution is available. So forgive me for being utterly sceptical that anyone will later disentangle all the code that has been comfortably internalized into the project.
With what is in Mirage right now -- but before creating a Mirage-specific shadow of most of opam -- disentangling the system seems about feasible.
Two major directions that couple everything together are:
The first point is really
The second point stands because we still don't know how to ship off the relevant backend-specific flags to the C compiler and linker. It's a case of cross-compilation, and this is routinely solved by setting up separate build environments, as opposed to packaging the cartesian product of build artefacts and targets.
If only we had a build tool that could pull this off.
Here's a data point: sequentially running
Especially compared to embracing the compatibility barrier.
I've opened a first set of PRs to switch mirage base packages to dune and use virtual libraries to have a nice mirage-os-shim common interface. This is important for mirage-entropy.
I haven't used Dune before. Currently, I am using the tag system of ocamlbuild to point that specific files are "precious" so as to allow certain *.cmi *.cmx files to be used by the unikernel, files that are generated by https://github.com/stedolan/malfunction
I hope that Dune has a similar feature.
At the moment, I am reading the mirage packages to implement a simple web server in the agda programming language. In general, you can use agda instead of OCaml to write a mirage unikernel.…
On Tue, May 28, 2019 at 7:16 PM Calascibetta Romain < ***@***.***> wrote: @xekoukou <https://github.com/xekoukou> really interesting, did you have an example of malfunction + mirage? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#969?email_source=notifications&email_token=AAG2FVGUVEJOQL6KGM33GULPXVLF7A5CNFSM4GWZ7RJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWMUZPY#issuecomment-496585919>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAG2FVGYYDS5CXV2VML7UKLPXVLF7ANCNFSM4GWZ7RJA> .
I still slightly fail to understand the high-level view, and connection between tools. afaiu duniverse is something to-be-integrated with mirage? I'd appreciate some remarks about the proposed workflows, both as a developer (where monorepo and no opam is fine) and as a CI system (where installing dependencies via opam is likely beneficial) -- also in respect to david's comments (will the
I've written a recap here https://gist.github.com/TheLortex/f3d92db831b553f6e4eaf2982e3e6427
What's interesting about having a
About the agda on mirageos use-case: these is coq support in dune so hopefully this shouldn't go wrong but I'll keep that in mind. Thanks for the examples link.
@hannesm I really hope that it's now clearer for you, but tell me if something is still missing.
@TheLortex thanks for writing this up, this makes the development workflow clearer. now, I'm a bit confused how a (non-interactive) CI flow should work -- is duniverse always required in the new model, or is it possible to only use opam for installing dependencies, and dune for building the unikernel?
The CI flow works the same way: duniverse is needed because packages with C stubs have to be locally compiled with the correct flags.
As in MirageOS 4, a repository can contain multiple
$ for c in `list all the config.ml`; do mirage configure -f $c <params_c>; done [..] # generates files, including opam and dune-workspace
Note: this doesn’t work fully yet, all the unikernels have to be configured to use the same target, but that will be fixed before the release.
As for MirageOS 3,
$ duniverse init [..] # read all the opam files, resolve depencies and create dune-get $ duniverse pull [..] # read dune-get and download all the dependencies in duniverse/
Question: do we want a
The main new feature of MirageOS 4 is that it is now possible to build all the unikernels in one go:
$ dune build @install [..] # generate all the unikernels in one go
$ dune exec -- solo5-hvt <path/to/unikernel.hvt> [..] # run the unikernel in `path/to`
Question: do we want to re-introduce a
In MirageOS 3,
Question: do we want to deprecate
In MirageOS 3,
pkg-config and ocamlfind predicates
In MirageOS 3, the compilation of C bindings were using pkg-config and an extra predicate in ocamlfind META files to replace, at link time, the default C archives by the one needed by the target. This is not the case in MirageOS 4 anymore.
Porting the runtime is a bit more tricky, as it shouldn’t be re-using the workspace CFLAGS that it is defining. To simplify things a bit, samoht/ocaml-solo5 is wrapping the build of solo5 and freestanding into one package per target. This allow all these packages to be co-installable.
#1153 now seems to work for