Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Dune instead of OCamlbuild to build unikernels #969

Open
TheLortex opened this Issue Feb 12, 2019 · 23 comments

Comments

Projects
None yet
5 participants
@TheLortex
Copy link

TheLortex commented Feb 12, 2019

I used several unikernel samples to explore what are the problems blocking the Dune transition. I've taken them from mirage-skeleton. After identifying problems and workarounds, I managed to build entirely with Dune + ld/pkg-config the following unikernels with unix, xen and hvt target: DNS, http_fetch, echo_server. I'm aware this does not cover everything but it's a good start.

Used features of OCamlbuild that aren't implemented in Dune:

  1. -dontlink <pkg> used for unix, threads, str and num. This removes these packages from the final linking step, thus allowing to link on non-Unix targets.
  2. Predicates is sometimes used for library selection.
  3. Use of -output-obj in conjunction with ..._linkopts in META files to link C stubs and custom OCaml runtime.

Workaround to each problem

  1. Explore the dependency graph to figure out which packages have unneeded dependencies and update them accordingly. Use OCaml 4.07 and Stdlib.Bigarray in order not to depend on the legacy bigarray which includes undefined stubs on xen/freestanding.
  2. Use virtual libraries and implementations. To not break everything, it's possible to keep the library name package as the default implementation and make it depend on the virtual library package-virtual. This way the update is transparent for users of the package and the ones that want to use the virtual library feature just have to change their package dependency into a package-virtual dependency.
  3. Dune uses -output-complete-obj to build object files. This also links C stubs and runtime. Therefore we don't have to care about C stubs anywore as they are automatically linked along with used libraries. No need for ..._linkopts flags in META. For the custom runtime, ocamlopt has a -runtime-variant option to choose a custom runtime library to link.

Packages to update

  1. ocplib-endian, mirage-fs-lwt, mirage-console-lwt, xen-evtchn, sexplib, tcpip, asn1-combinators, mirage-entropy
  2. mirage-entropy, zarith + dependencies (asn1-combinators, nocrypto)
  3. mirage-xen-ocaml, ocaml-freestanding: just need to add a symlink from lib...asmrun.a to ocaml runtime directory so that ocaml finds the runtime variants. I've already modified these packages locally to make my experiments so it's not that hard to make these changes.

Issues and comments

  1. Duniverse forks may have already fixed some of these packages.
  2. The uses of virtual libraries disable cross-module optimisation such as inlining. We have to be careful with that.

How do you feel about this ? How should we proceed to update packages ? Am I missing something ? Feel free to comment !

@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 12, 2019

This plan sounds good to me!

About specific libraries:

@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 12, 2019

pushed xen-evtchn pr to move to dune

@TheLortex

This comment has been minimized.

Copy link
Author

TheLortex commented Feb 12, 2019

  • ocplib-endian: in opam file, cppo is not written as a build dependency
  • mirage-*-lwt: in opam file, unnecessary requirement of cstruct-lwt
  • xen-evtchn: in lib/dune:4, dependency on unix and bigarray
  • mirage-entropy: will push that on dune-universe !

avsm added a commit to dune-universe/ocplib-endian that referenced this issue Feb 12, 2019

TheLortex added a commit to dune-universe/mirage-entropy that referenced this issue Feb 12, 2019

TheLortex added a commit to dune-universe/mirage-entropy that referenced this issue Feb 12, 2019

@avsm avsm referenced this issue Feb 12, 2019

Open

[WIP] Dunify with variants #384

0 of 6 tasks complete
@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 12, 2019

I pushed a rebase of mirage/mirage-tcpip#384 now to use virtual_modules -- feel free to push to that branch as well with any fixes

avsm added a commit to avsm/ocaml-evtchn that referenced this issue Feb 12, 2019

avsm added a commit to avsm/mirage-console that referenced this issue Feb 13, 2019

TheLortex added a commit to TheLortex/mirage that referenced this issue Feb 13, 2019

TheLortex added a commit to dune-universe/ocaml-asn1-combinators that referenced this issue Feb 13, 2019

TheLortex added a commit to dune-universe/mirage-entropy that referenced this issue Feb 13, 2019

TheLortex added a commit to dune-universe/ocaml-nocrypto that referenced this issue Feb 13, 2019

Zarith is now virtual
For mirage/mirage#969
Add lwt/mirage-entropy dep

TheLortex added a commit to dune-universe/mirage-tcpip that referenced this issue Feb 13, 2019

Some fixes for tcpip
For mirage/mirage#969
`implements` requires public library name.
Dummy tcpip lib so that `tcpip` is resolved.
Fix dependencies in the opam definition.
@TheLortex

This comment has been minimized.

Copy link
Author

TheLortex commented Feb 13, 2019

After struggling with bigarray today I finally opted in for a hack: even if ocamlopt links caml_ba_map_file and adds some unresolved symbols we can choose to ignore it.
My Mirage CLI fork is available on TheLortex/mirage. I needed some local opam pins to get http_fetch compiling on every target (except genode because of dynlink issues).
It's not that much ! Most problems are actually solved by taking the duniverse version of projects.

  • mirage cli
  • mirage-entropy + variants
  • nocrypto
  • tcpip
  • tcpip-checksum + variants
  • TLS: tls, tls-lwt, tls-mirage
  • Zarith: zarith, zarith-freestanding, zarith-xen, zarith-virtual
@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 13, 2019

We might as well start releasing some of these... I think tcpip is a good one, and then a zarith-mirage fork.

TheLortex added a commit to dune-universe/mirage-tcpip that referenced this issue Feb 14, 2019

Some fixes for tcpip
For mirage/mirage#969
`implements` requires public library name.
Dummy tcpip lib so that `tcpip` is resolved.
Fix dependencies in the opam definition.

TheLortex added a commit to TheLortex/mirage-platform that referenced this issue Feb 14, 2019

Link and C flags are also specified in dune-accessible files.
For mirage/mirage#969
Instead of pkg-config, one can use the following files to get the
compilation flags:

ocaml-freestanding/libs
ocaml-freestanding/cflags

TheLortex added a commit to TheLortex/mirage-platform that referenced this issue Feb 14, 2019

Add library flags as files.
For mirage/mirage#969
Instead of PKG-CONFIG, one can use the following files to get
compilation flags:

mirage-xen-ocaml/libs
mirage-xen-ocaml/cflags
mirage-xen-posix/minios-cflags
mirage-xen-posix/minios-libs
mirage-xen-posix/posix-cflags
mirage-xen-posix/posix-libs

TheLortex added a commit to dune-universe/mirage-entropy that referenced this issue Feb 14, 2019

Get rid of pkg-config + new package definitions
For mirage/mirage#969
Along with:
mirage-platform: 0612bdbfe1eeb004ad923991178256feb780d14c
ocaml-freestanding: c5bf86ce29872b65a12ac42b9104f15f063d644e

Instead of using pkg-config, cflags and libs are fetched from files in
the library folder.
@TheLortex

This comment has been minimized.

Copy link
Author

TheLortex commented Feb 14, 2019

New update for mirage-entropy:
Instead of using pkg-config to fetch parameters, we can take advantage of dune's %{lib:..} variable expansion and :include to gather C flags. This technique can also be used for every package that builds xen/freestanding stubs.

@hannesm

This comment has been minimized.

Copy link
Member

hannesm commented Feb 14, 2019

@TheLortex thanks for your work on this. Yes, a good outcome would be to also remove pkg-config from our toolchain. :)

@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 14, 2019

Looks good -- could you do a PR of mirage-platform that adds the files, @TheLortex? We can release that independently since it can remain backwards compatible.

@rgrinberg

This comment has been minimized.

Copy link
Member

rgrinberg commented Feb 14, 2019

  • Use virtual libraries and implementations. To not break everything, it's possible to keep the library name package as the default implementation and make it depend on the virtual library package-virtual. This way the update is transparent for users of the package and the ones that want to use the virtual library feature just have to change their package dependency into a package-virtual dependency.

As discussed before, I feel like this should be a first class feature in dune. Making existing libraries virtual is something that is always going to come up if this feature gains any traction. Dune should properly support such transitions instead of having the users pollute the package namespace.

@ehirdoy

This comment has been minimized.

Copy link

ehirdoy commented Feb 15, 2019

Any plan to support Bytecode in MirageOS?

@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 15, 2019

@ehirdoy that question appears irrelevant to this issue. It would be helpful to create a new issue with your query. When doing so, please also specify what you want to accomplish with bytecode to provide more context.

avsm added a commit to avsm/opam-repository that referenced this issue Feb 19, 2019

+mirage-platform.3.3.0
supports forthcoming dune support as part of mirage/mirage#969:

* Add support for OCaml 4.08 (mirage/mirage-platform#206 by @anmonteiro)
* `libxenasmrun.a` is symlinked in the ocaml directory in order to be able to use ocamlopt's `-runtime-variant` option.
* Expose flags through files as well as pkg-config (mirage/mirage-platform#205 by @TheLortex).  Instead of pkg-config, one can use the following files to get compilation flags:
  * mirage-xen-ocaml/libs
  * mirage-xen-ocaml/cflags
  * mirage-xen-posix/minios-cflags
  * mirage-xen-posix/minios-libs
  * mirage-xen-posix/posix-cflags
  * mirage-xen-posix/posix-libs
  With dune this allows us to write `%{lib:mirage-xen-posix:posix-libs}` to get the flags instead of having a script invoking `pkg-config mirage-xen-posix --libs`.
@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 20, 2019

mirage-platform released to opam with the flags now

@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 20, 2019

@hannesm is going to look at the dune release of tls and friends. I'm taking a look at the zarith fork now

@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 20, 2019

the zarith dune fork is now tested in CI on macOS, Windows, Linux (amd64/arm64) and passes tests https://dev.azure.com/avsm0653/zarith/_build/results?buildId=101

@TheLortex do you have a repo of Zarith with the various freestanding/xen/virtual variants at the moment? The dune-universe fork only has the core library.

Also, I think we need to add gmp to the fork list, as it also needs freestanding variants.

@TheLortex

This comment has been minimized.

Copy link
Author

TheLortex commented Feb 20, 2019

I'm working on fixing mirage-platform and ocaml-freestanding.
For Zarith, I created a variants branch in dune-universe (https://github.com/dune-universe/Zarith/tree/variants).
What do you think is the best ? One package for each variant (zarith, zarith-virtual, zarith-xen, zarith-freestanding) or one big packages with subpackages (zarith, zarith.virtual, zarith.xen, zarith.freestanding) ? (optional implementations were broken ocaml/dune#1856 but this is now fixed)

@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 22, 2019

One package for each variant (zarith, zarith-virtual, zarith-xen, zarith-freestanding) or one big packages with subpackages (zarith, zarith.virtual, zarith.xen, zarith.freestanding) ?

A big part of making this design work is to ensure that the dependencies on exotic backends are optional, precise and coinstallable. With opam depopts, they were optional but not precise or coinstallable (since making a backend available required installing multiple opam packages which would change the behaviour of each other). The current situation also means they are not coinstallable, since solo5 and xen packages conflict in opam.

So with the variants, I'd recommend making one opam package per variant so that they can be cleanly installed separately. However, I'm not sure how to avoid having to specify every single variant in every single package that has C bindings. It would be amazing if we could somehow centrally specify a new backend (e.g. esp32) and have opam packages spring up in each of the libraries that have variants for those new options (e.g. zarith-esp32) without having to go through and manually add it to each one.

@hannesm

This comment has been minimized.

Copy link
Member

hannesm commented Feb 22, 2019

The current situation also means they are not coinstallable, since solo5 and xen packages conflict in opam.

the reason is mainly that they both modify zarith's META file (adding some linkopts in there).

zarith, zarith-virtual, zarith-xen, zarith-freestanding

sorry, I'm somehow missing the actual design here. as it works now, the zarith META file (and others) are extended with xen/freestanding_linkopts predicates that are picked up by ocamlbuild during the link step to link in the required C libraries.

what should the new approach look like? I understand that dune understands variants (i.e. checkseum.c / checkseum.ocaml, but also zarith / zarith-xen / zarith-freestanding). now: when and how are the platform-specific libraries (opam packages) picked up? (and in addition, for checkseum (+digestif etc.) who'll select whether the c or the ocaml implementation is to be used?).

and what is zarith-virtual? is this the package which defines "for freestanding, use zarith-freestanding; for xen, use zarith-xen; etc."?

are these variants open, i.e. can anyone extend them (for a new target, etc.)? can there be conflicts (certainly digestif.ocaml and digestif.c are conflicting, and there's no obvious solution, apart from asking the unikernel developer (or operator) to decide), and how does conflict resolution work there?

sorry for being confused by that, I also don't understand these virtual libraries -- is this an opam feature, or dune feature? in opam there was at some point a proposal to have virtual packages, but i don't think it is implemented since it's pretty hard to have if you take security into consideration. it seems like i'm missing some concrete design document here.

@TheLortex

This comment has been minimized.

Copy link
Author

TheLortex commented Feb 25, 2019

sorry, I'm somehow missing the actual design here. as it works now, the zarith META file (and others) are extended with xen/freestanding_linkopts predicates that are picked up by ocamlbuild during the link step to link in the required C libraries.

what should the new approach look like? I understand that dune understands variants (i.e. checkseum.c / checkseum.ocaml, but also zarith / zarith-xen / zarith-freestanding). now: when and how are the platform-specific libraries (opam packages) picked up? (and in addition, for checkseum (+digestif etc.) who'll select whether the c or the ocaml implementation is to be used?).

Packaging is basically the same. Variants allow dune to select implementations at build time. Instead of xen/freestanding_linkopts, all the linking data is contained in the cmxa's that are generated when the Zarith libraries are built. This also resolves the META file conflict. C/OCaml implementation can either be selected by the user using the variants option that will be added in Dune, and the library developer can choose a default implementation. If in the end there's ambiguity, an error message will be shown and the user will have to choose.

and what is zarith-virtual? is this the package which defines "for freestanding, use zarith-freestanding; for xen, use zarith-xen; etc."?

zarith-virtual is the package that defines the Zarith interface that variants have to implement. It may also define a default implementation if no variants are selected.

are these variants open, i.e. can anyone extend them (for a new target, etc.)? can there be conflicts (certainly digestif.ocaml and digestif.c are conflicting, and there's no obvious solution, apart from asking the unikernel developer (or operator) to decide), and how does conflict resolution work there?

The design makes variants open, thus anybody can extend them. There can be conflict if there is no default variant, or if two libraries implement the same library with the same variant. In this case the user will have to choose by specifying one of the libraries as a direct dependency.

sorry for being confused by that, I also don't understand these virtual libraries -- is this an opam feature, or dune feature? in opam there was at some point a proposal to have virtual packages, but i don't think it is implemented since it's pretty hard to have if you take security into consideration. it seems like i'm missing some concrete design document here.

Virtual libraries is a dune feature. You can read about it here: https://dune.readthedocs.io/en/latest/variants.html
Variants is the mechanism that allows to select implementations using a set of tags. This is not implemented yet but it's in project.

I hope this made everything a bit clearer :)

@hannesm

This comment has been minimized.

Copy link
Member

hannesm commented Feb 25, 2019

thanks, yes, I've a better picture in my head now :)

@avsm

This comment has been minimized.

Copy link
Member

avsm commented Feb 25, 2019

To clarify the very high level picture, with this dune building mode both opam and pkg-config are optional to build a unikernel. Once you have a 'closed duniverse' (that is, all the ocaml libraries in one directory structure with no external dependency cycles to system/opam-installed libraries), then that can build with a single dune build and no other tooling. All mirage backends are also simultaneously buildable in a single directory structure as variants ensure that full hygiene is maintained wrt cross-compilation for different backends.

This allows all code for a unikernel to be checked into a git repository and be highly reproducible. From there, we can introduce opam to publish/pull slices of the directory structure, and pkg-config to do top-level configuration (e.g. for directory search paths for compilers or similar).

@hannesm

This comment has been minimized.

Copy link
Member

hannesm commented Mar 17, 2019

since some packages are now merged + upstreamed (notably mirage-xen-ocaml / ocaml-freestanding) with an unclear status of how that should work, I'd appreciate a branch on https://github.com/mirage/mirage-dev that includes a universe of packages which work nicely together with this using-dune-for-build approach before further PRs and releases are done. Mainly to avoid more opam repository release churn for things that can be avoided, such as the ocaml-system conflict. Thanks again for working on this! This way, the (mirage-dev travis) CI will compile mirage-skeleton for different targets.

TheLortex added a commit to TheLortex/mirage that referenced this issue Mar 18, 2019

@TheLortex

This comment has been minimized.

Copy link
Author

TheLortex commented Mar 18, 2019

I started doing that in a dune branch in mirage-dev. I haven't tested all targets and samples but it should at least build for simple ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.