Dunification of MirageOS

[dinosaure]

cc @mirage/core

This is the final proposal about the dunification of MirageOS. Some specific tasks which are discussed but not yet implemented into this PR:

• remove sexplib
• support of Xen
• delete libdir
• an option to use others variants

compatibility and flags

-output-obj

MirageOS in the current version trusts on one command-line argument with ocamlopt: -output-obj. As I explained to @mato, this flag asks to ocamlopt to produce a main.o (from main.cmx generated by functoria) without any link step.

Then, the mirage tool collects some information with ocamlfind and retrieves static libraries needed with linkopts_freestanding or linkopts_xen. About the unix target, it follows surely flags given by *.cmxa. However, dune wants to use for several reasons (eg. -output-obj on ocaml/ocaml#6797) the new option -output-complete-obj which will follow flags given by *.cmxa.

The problem about the link-step is on this new option. We can not continue to link *.cmxa together and then, in another link step, choose right static libraries compiled for the given target. So, at least, we have two solution:

• do by ourselves the link-step
• provide well-formed *.cmxa with right flags (-L and -l)

Into #1020/#1021, the choice was made to continue to trust on ocamlfind and use -noautolink to mimic the behavior of -output-obj. In another, the choice was made to continue to resolve static libraries with ocamlfind (with given META files) and then, do a translation between C flags given by ocamlfind/META/pkg-config to ocamlopt flags (eg. wrap -L with -ccopt and -l with -cclib).

This solution is not very good when we add a new tool (cc-to-opt) and discard some static libraries (with -noautolink) if they are not noticed by the META file (eg. digestif/checkseum).

The second solution was chosen to trust on the orchestration of dune. But even if it's what we want, I just would like to compare what it should happen with this PR and what we did before. The choice of dune is, in many ways logic and expected by a consensus of all @mirage/core and the OCaml community. But this same choice was made about ocamlfind a long time ago. I mean, we do exactly what we did before, we will trust on an external tool of the MirageOS project about a particular step (which is not present in many projects).

I don't want to say that dune is may be the wrong choice but I would like to say that it implies a strong follow about what happens inside dune. Currently, and from what I saw with @rgrinberg and @diml, the feedback-loop is very good - so I don't have anything to blame. But, please, keep in your mind that you should keep this state: dune + mirage = ❤️ as long as possible.

-no-keep-locs

Since OCaml 4.06.0, -keep-locs is the default option when we compile *.cmi. For a long time and across several libraries from the MirageOS eco-system (and in many ways), we trusted on the fact that a *.mli can be compiled to a *.cmi and the resulted hash should be reproducible despite the location of the compilation computation.

However, it's not true now and for a long time but I'm not sure that all the MirageOS stack followed this update (eg. #1005). It's currently a problem when some libraries still continue to follow this old rule when they expect to plug at the link time an implementation according an interface: in other words, the linking-trick.

Of course, dune comes with a solution, modules_without_implementation and variant. But, again, it's true that many strong pieces of the MirageOS stack did not follow this update (eg. the dunification) too. At the end, it still exists some tricks into our stack and it seems hard to just remove them. So, this PR wants to deal with this, again, with the smallest patch as we can.

Intermediate solution

I would like to say that this solution is not fixed. It's an intermediate solution where the move the MirageOS stack needs much more work than what I can alone. Some required updates are well known but the time needed to ship a release of these projects is not.

So, instead to wait that some projects are updated in our way, this PR wants to provide a reasonable solution first for all people. Some points are not perfect but, as I said into #1020, we can start to figure out about details/features/etc. only if we have an dunified MirageOS.

Deletion of pkg-config

In the way of #1020, #1021 and #979, we still continue to uses some files provided by the platform instead to use pkg-config inside the mirage tool. We still continue to use pkg-config on these platforms (eg. ocaml-freestanding and mirage-posix-xen) to craft these flags, however.

These flags are:

• libs.sexp which contains path to static libraries:

  • libasmrun.a
  • libnolibc.a
  • libopenlibm.a

  and an explicit link to these libraries. For Solo5, it comes with libmirage-solo5_bindings.a which has some needed symbols. This file is used by the first link step. with -output-complete-obj.

• ldflags which contains flags needed to use ld at the final link step.
  The link step is about Solo5 and Xen when we need to follow a linking-script given by the requested platform to produce the unikernel

• cflags.sexp which contains right flags needed to compile any C files. These flags are used by the Solo5 target to compile the manifest.c generated by solo5-elftool with the manifest.json and used by the final link step.

• libdir is provided but not used anymore.

This solution seems to be the right and @mirage/core agree to delete pkg-config from the mirage tool. Again, a better solution can be found (a pkg-config in OCaml?) but I think, from the OCaml ecosystem point-of-view, we don't need more that what these files give to us.

The support of Xen is not done but should follow the same pattern. I will try to fix that this week but only if people agree with this last proposal - I mostly want to save my time.

Deletion of ocamlfind

A deletion of ocamlfind is much more complex that what we believe when we followed the last layout about ocamlfind/META files on several projects including some outside the MirageOS organization.

As I said, dune comes with a solution and even if I can swear that some of these projects will move to dune and will use variant as we expect, some projects will not follow this move for many reasons.

Currently, some neuralgic projects must be updated with this new layout:

• nocrypto
• digestif
• checkseum
• bigstringaf (cc @seliopou)

I'm sure that I forgot some others projects, so help me on this list.

The point is to use variant. The idea is to have an interface which has all external. Then, the developer should provide 3 implementations of this interface:

• an unix one, basic compilation of C stubs
• a freestanding with a compilation of C stubs with flags given by ocaml-freestanding/cflags.sexp
• a xen with a compilation of C stubs with flags given by mirage-posix-xen/cflags.sexp

These implementations must denote a variant with: (variant {unix,freestanding,xen}) and to avoid any link error for others users, denote a default implementation (the unix one). Then, the mirage tool will generate a dune.build which will select rights variants.

In the case of checkseum or digestif when we need variants to provide unix, freestanding and xen stubs, we should be able to specify an other variant to use the OCaml implementation of the C implementation.

In the case of nocrypto, the story is much more complex than what we think. nocrypto uses C stubs (as digestif) and a virtual *.cmi, in our case: Z.cmi. This module should be provided by the Zarith distribution. However, this distribution comes with another static libraries of C files which should be compiled with right flags. The problem about Z.cmx can also be found by:

• asn1-combinators
• x509

At this stage, I suspect that nocrypto still trust on the initial behavior of *.cmi with -no-keep-locs.

A fork of nocrypto

#1020 and #1021 want to keep the compatibility with nocrypto.0.5.4-2 and avoid any needed works on this project when its future is not clear. However, the solution is not good as some people said (including me). The solution was to do what the mirage does currently but the diff is which command will eat our link flags? Currently, the mirage tool gives flags of static libraries to ld where, with dune, we must give them to ocamlopt.

Then, nocrypto aggregates at the end all of our problems. And when some can be fixed without any change of nocrypto (eg. #1005, #1009 and #1010), I think we reach the end of the current status of nocrypto when we must update it, at least, make a new release of it.

A work was done on mirleft/ocaml-nocrypto#158 about the dunification but it's not merged.

So I did a fork of nocrypto available here: https://github.com/dinosaure/ocaml-nocrypto/commits/dune from mirleft/ocaml-nocrypto#158 (I need to clear the history and I will make a description of each commit in another comment). I want to update the layout of the library according dune, it fixes some bugs about endian.h and it provides, as expected, nocrypto.native.{unix,freestanding,xen}.

This version wants to stay compatible with the current version of TLS. So it includes some ppx on some types.

At the end, this version of nocrypto will do the right orchestration about which zarith we want according variants and it uses only Z.cmi as an interface of libgmp. As we believe, dune will do the right orchestration according variants/target and it will choose the right nocrypto.native which depends on the right zarith implementation.

Again, the solution is not the best where we completely fork a project to unlock the dunification of MirageOS. But, as @mato said, nocrypto is a part of an important stack of MirageOS and it was not released since 2017. We can not move forward without it but we can not continue to use it as is. From what I know, @avsm did a good mediation about it but it seems clear for me (and from others) that we should make a decision.

Patches needed for others projects

The new repository mirage-dune available here: https://github.com/dinosaure/mirage-dune wants to provide the best way to move forward and include all needed patches for others projects to compile and use them with this PR.

Zarith

The most important patch and the smallest is about zarith. As I said, we need to provide a way to plug an implementation under Z.cmi but we should ensure that Z.cmi produced and used by nocrypto/x509/asn1-combinators still is the same (the same hash). As a reasonable a patch, we should put -no-keep-locs when we want to build Zarith and do a release of zarith.1.7-1 with this patch. The distribution does change, the code does not change, just an option is added.

Of course, we can start to think about a better solution like: dunify Zarith. But:

1. We don't want to fork the world
2. We don't want to wait a possible release of it with dune

So, this is the most reasonable as I can and I believe that we can do a release of it as soon as possible.

From it, a patch is needed about zarith-freestanding (and xen) when the provided *.cmxa of these packages should link with the right GMP library:

ocamlmklib put by default -lzarith into zarith.cmxa. By this
way, any link with this zarith.cmxa, even if we compiled it with
right flags requested by ocaml-freestanding, will link with
libzarith.a (compiled by default flags).

In our case, we want to link with -lzarith-freestanding according
what mirage-install.sh does - the shell script copies libzarith.a
to libzarith-freestanding.a. So we add a new option -oc to tell
to ocamlmklib to link with -lzarith-freestanding correctly.

Then, mirage-install.sh copies zarith.cmxa to zarith-freestanding.cmxa
and add a new META information. zarith-freestanding.cmxa will be
chosen if we want to link with zarith.freestanding.

--- a/project.mak	2017-10-13 19:45:41.000000000 +0200
+++ b/project.mak	2018-11-30 19:11:16.205189144 +0100
@@ -60,10 +60,10 @@
 	make -C tests test
 
 zarith.cma: $(MLSRC:%.ml=%.cmo)
-	$(OCAMLMKLIB) -custom -o zarith $+ $(LIBS)
+	$(OCAMLMKLIB) -custom -o zarith $+ -oc zarith-freestanding $(LIBS)
 
 zarith.cmxa zarith.$(LIBSUFFIX): $(MLSRC:%.ml=%.cmx)
-	$(OCAMLMKLIB) -custom -o zarith $+ $(LIBS)
+	$(OCAMLMKLIB) -custom -o zarith $+ -oc zarith-freestanding $(LIBS)
 
 zarith.cmxs: zarith.cmxa libzarith.$(LIBSUFFIX)
 	(OCAMLOPT) -shared -o $@ -I . zarith.cmxa -linkall

This patch was tested with the great CI of OPAM (eg. ocam/l/opam-repository#15522) and should not break anything with the current state of MirageOS - where initially we link with zarith.cmxa.

asn1-combinators and x509

Then, we need to upgrade projects which want to use Z.cmi and mostly still continue to use -no-keep-locs and import Z.mli from Zarith and compile it as a virtual interface of an implementation of the module Z. Again, the orchestration will be done by nocrypto, at this stage, use x509 or asn1-combinators with this patch will produce a link error where they will expect an implementation (as digestif before).

I provided:

• asn1-combinators.0.2.0-1 which wants to apply the patch
• asn1-combinators.0.3.0 which took the current trunk of asn1-combinators and put the patch
• x509.0.8.1-1 which wants to apply the patch

tls

At the end, of course, we have tls but the only patch I did about this project is to give the right implementation when we want to compile tests. Of course, again, we can start to think about a dunification of this project but the current state of tls is good enough to support a dunification of MirageOS. I mean, the bottleneck is not here. So I provided tls.0.10.5-1 where the patch, I think, is not needed for an usual installation (but I'm not sure).

sexplib and ocplib-endian

These libraries are already noticed in #1020 and #1021:

• A pull-request was already done about sexplib: Use bigarray-compat instead bigarray to avoid a link with unix janestreet/sexplib#40
• I did two packages for ocplib-endian:
  • ocplib-endian.1.0-1 which includes a patch for bigarray-compat with the current release of this package
  • ocplib-endian.1.1 which took the current trunk and apply the patch about bigarray-compat

Conclusion

From what I know, I can compile mirage-skeleton with this PR and all of packages available into dinosaure/mirage-dune. Again, others solutions can exist but a decision should be done as soon as possible to go forward, merge it an fix rest of issues then.

As I said into #1020 and #1021, many possibilities exists then about features given by dune but all need a dunification of MirageOS first.

Other solutions than dune exist too when I mentioned a possible way to do by ourselves the link step but we need a decision from all @mirage/core about next. So I just would like to say that it will be my last proposal about the dunification according expected results from all people.

Then, if all people are agree with this pull-request and whet it implies about our ecosystem (in a general way), I will continue the support of Xen, merge it and then, do the support to move others libraries to this new layout. Thanks for reading.

[dinosaure]

I just would like to clarify some underlying requests about this PR to understand choices made as long as #979, #1020 and #1021. From all people, requests in my mind are exhaustive such as:

• properly do the dunification of MirageOS
• a way to keep a compatibility with the OPAM ecosystem
• a way to be compatible with the duniverse tool
• the smallest update to reach all previous requests

From that, I followed the work did by @TheLortex who focused his work on the usability of duniverse into a unikernel workspace. This PR is on top of that and even if I did not mention the compatibility with duniverse, this fact still is valid in this PR - eg. expansion of right C flags into the dune context.

However, I took care about some others details mostly to avoid an hard incompatibility of existing resources and give a chance to people to switch from the current mirage tool to the new one (when the new one is not fully formalized and known to all).

By this fact, I keep in mind and in this PR the layout provided by OPAM where the MirageOS ecosystem trusts. The hard question from all of that is how to deal with both worlds as an intermediate step. From what I saw, all people consider it as an intermediate step which is not perfect but let a window for all of us to move forward and be synchronized properly together in one fully-define goal.

The fact that the idea behind dune is not clear for everybody when we never take the time to formalize layout of libraries. Some people want something which just works to be able to start to think about next step and this PR serves this purpose. It seems that some other have already an idea about what it should happen next, but they need dune to unlock the ability to experiment their ideas.

This PR is not a perfect way to switch MirageOS to dune because it's a consensus which, I believe, is reached. And I always opened a door for any concerns. So if I miss something, firstly sorry, secondly you should start to write a comment.

[dinosaure]

nocrypto

As I said, I will explain my patch about nocrypto. As I said, my work is on top of mirleft/ocaml-nocrypto#158. From it, I added 14 commits which are well defined:

• Unlock variants: it complete dune-project to support variants
• Surround endian.h with C macro: as noticed here, this patch wants to prevent a cyclic inclusion
• Add a freestanding implementation of endian.h: ocaml-freestanding does not give to us some convenience functions on that, I re-implemented them according __BYTE_ORDER
• Move C stubs to and other directory: this patch help me to copy C stubs to others directories which will be specific to targets
• A new virtual library nocrypto.native: no_native is only an interface into the core library now
• Delete C stubs from the core library: a clean of the dune file to compile as usual an OCaml library
• Redo ppx_sexp_conv: to stay compatible with tls.0.10.5 which expects some sexp converters
• Add unix implementation
• Add freestanding implementation
• Use bigarray-compat instead bigarray
• Z as a virtual library: we copy z.mli and compile it with -no-keep-locs, then the orchestration of nocrypto.native will come with the right Zarith library as explained before according the requested target
• Use dune-configurator instead dune.configurator
• Fix tests according the new layout of nocrypto

I pushed-force into https://github.com/dinosaure/ocaml-nocrypto/tree/dune, so feel free to review and comment.

The plan to dunify MirageOS

As I said before, a consensus must be found with all parts to go further about the dunification of MirageOS. I think, I did not forget any actors of the MirageOS project to make this PR. So, if I'm wrong, please let a comment.

Now, some parts are missing in this PR but the plan is clear and the expected layout is well documented for all people. So will ask an agreement of all @mirage/core before to continue this PR. If we reach a state where all people accept ideas/layout/design behind this PR, I will be able to do anything to merge this. If this agreement is not reached, I will close this PR.

Thanks for reading!

[samoht]

can you write all of these lines in one big {sexp|...|sexp} statements instead of splitting in multiple lines?

[samoht]

same here, I find it easier to maintain/review if there is one main file template instead of splitting this into multiple statements.

[samoht]

The patch looks good to me. To make it even simpler, I think it would be nice:

• remove the dependency to sexplib (and use Fmt.pr directly)
• try to "template" the dune files as much as possible, e.g. putting everything into one main template instead of mixing fragments as it's a bit harder to see the final shape of the file in one glance
• remove dead-code instead of commenting it

And I am very much in favour for using dune variants, as it will simplify a lot of things.

For the C bindings: a possible solution to simplify the ports to MirageOS that we discussed last year would be to provide a tool to generate the correct dune files to be mirage-compatible (e.g. the equivalent of ocb-strubblr but for dune). That tools will know about the existing backends (so will automatically generate the correct runes in case of a new backend) and it could ease upstream adoption by generating a minimal patch and no maintenance cost.

[dinosaure]

sexplib is deleted now. Even if it's good to avoid a huge dependency, we lost any way to check that {sexp|...|sexp} is well formed.

[dinosaure]

Currently, Xen support was done but I'm not able to reproduce anything when Travis disappeared and I don't know how to use OCaml CI when it stuck on "Unable to parse .ocamlformat file". Do we have a way to use OCaml CI with this PR correctly?

[samoht]

You need to merge with master (#1026 has a fix for the malformed ocamlformat file).

[dinosaure]

As the new year, Travis CI is happy and OCaml CI is happy. So the PR is ready to merge according the OPAM repository available here: https://github.com/dinosaure/mirage-dune.git.

I did not test Xen unikernels but they should work. According what I did:

• we need to apply some few patches mostly about
  • asn1-combinators
  • x509
  • tls
• we need to fork nocrypto
• we need to apply some patches into zarith and do a release about zarith-{freestanding,xen}
• we need to do a release on mirage-xen, ocaml-freestanding and mirage-platform

So, I will prepare all of these PRs next week. Happy new year 🎉 !

[samoht]

Maybe cleaner to just remove that function?

[dinosaure]

After discussion, zarith should be dunified too to avoid patch on x509 and asn1-combinators. It should provides xen and freestanding back-end too. So I add a new goal on this PR:

• dunify zarith

The way to upstream the dunification of zarith is not clear however. Works still will be available into dinosaure/mirage-dune. According this goal, ocaml/opam-repository#15522 can be closed where we should properly use dune's variants.

[samoht]

Closing in favour of #1153 ; thanks for the hard work it was very useful to have that PR as a starting point of that work!