Use Dune instead of OCamlbuild to build unikernels #969
Comments
|
This plan sounds good to me! About specific libraries:
|
|
pushed xen-evtchn pr to move to dune |
|
|
I pushed a rebase of mirage/mirage-tcpip#384 now to use virtual_modules -- feel free to push to that branch as well with any fixes |
For mirage/mirage#969 Add lwt/mirage-entropy dep
For mirage/mirage#969 `implements` requires public library name. Dummy tcpip lib so that `tcpip` is resolved. Fix dependencies in the opam definition.
|
After struggling with bigarray today I finally opted in for a hack: even if ocamlopt links
|
|
We might as well start releasing some of these... I think tcpip is a good one, and then a zarith-mirage fork. |
For mirage/mirage#969 `implements` requires public library name. Dummy tcpip lib so that `tcpip` is resolved. Fix dependencies in the opam definition.
For mirage/mirage#969 Instead of pkg-config, one can use the following files to get the compilation flags: ocaml-freestanding/libs ocaml-freestanding/cflags
For mirage/mirage#969 Instead of PKG-CONFIG, one can use the following files to get compilation flags: mirage-xen-ocaml/libs mirage-xen-ocaml/cflags mirage-xen-posix/minios-cflags mirage-xen-posix/minios-libs mirage-xen-posix/posix-cflags mirage-xen-posix/posix-libs
For mirage/mirage#969 Along with: mirage-platform: 0612bdbfe1eeb004ad923991178256feb780d14c ocaml-freestanding: c5bf86ce29872b65a12ac42b9104f15f063d644e Instead of using pkg-config, cflags and libs are fetched from files in the library folder.
|
New update for |
|
@TheLortex thanks for your work on this. Yes, a good outcome would be to also remove pkg-config from our toolchain. :) |
|
Looks good -- could you do a PR of mirage-platform that adds the files, @TheLortex? We can release that independently since it can remain backwards compatible. |
As discussed before, I feel like this should be a first class feature in dune. Making existing libraries virtual is something that is always going to come up if this feature gains any traction. Dune should properly support such transitions instead of having the users pollute the package namespace. |
|
Any plan to support Bytecode in MirageOS? |
|
@ehirdoy that question appears irrelevant to this issue. It would be helpful to create a new issue with your query. When doing so, please also specify what you want to accomplish with bytecode to provide more context. |
|
I'm trying to argue that this is making a bad situation worse. Because even right now, Mirage is a monorepo project. In fact, here is something missing from https://mirage.io: MirageOS is not, in general, compatible with third-party OCaml code. Users cannot expect separately developed OCaml libraries to work in the unikernel environment, unless and until they are accepted into the MirageOS project and ported accordingly. And while this could be phrased a little less directly, it cannot be made more true. It's not even by design; this is just a consequence of long-standing technical debt. Just to put everyone on the same page, please bear with me for a moment, while I try to give some context: Up until 2015 this was quite obvious. For example, there was a copy of As more C came into play, it progressed to the next stage -- in order to run in a unikernel, every library had to build its stubs for all mirage targets and broadcast its intentions to the This gave us several From there, we've made the first steps towards making this all-in-one approach a formal reality. Which brings us to the present. The current proposal is a perfectly logical consequence of bringing everything that could possibly be compiled into a unikernel under one roof (directly, by forking, or through an overlay). With everything building from a single project, there is nothing odd about a compilation scheme which requires
because what is being proposed here is understood as a Mirage-internal refactoring. As far as I can tell, the current state of Mirage is to completely embrace this idea that separate development is off the table. If everything is not understood as simply being part of Mirage, then this scheme is so prohibitively invasive to the point of being anti-modular. Now, I really am glad that we can agree on the ultimate vision, but we can probably all agree on what we want to have at the point at infinity. Namely, everything. What is more interesting is whether the next step will
If it increases, it builds a larger wall around Mirage. And if history is any guideline, people stop working on a problem the very moment any solution is available. So forgive me for being utterly sceptical that anyone will later disentangle all the code that has been comfortably internalized into the project. The punchline? With what is in Mirage right now -- but before creating a Mirage-specific shadow of most of opam -- disentangling the system seems about feasible. Two major directions that couple everything together are:
The first point is really The second point stands because we still don't know how to ship off the relevant backend-specific flags to the C compiler and linker. It's a case of cross-compilation, and this is routinely solved by setting up separate build environments, as opposed to packaging the cartesian product of build artefacts and targets. If only we had a build tool that could pull this off. 👀
Here's a data point: sequentially running Especially compared to embracing the compatibility barrier. |
|
Regarding this error:
Pointed out by Hannes in #969 (comment) Could anyone clarify where is this path read from? @avsm, @yomimono or @TheLortex perhaps? EDIT: I have fix a for the above: ocaml/dune#2124 |
|
I've opened a first set of PRs to switch mirage base packages to dune and use virtual libraries to have a nice mirage-os-shim common interface. This is important for mirage-entropy. |
|
|
I haven't used Dune before. Currently, I am using the tag system of ocamlbuild to point that specific files are "precious" so as to allow certain *.cmi *.cmx files to be used by the unikernel, files that are generated by https://github.com/stedolan/malfunction I hope that Dune has a similar feature. |
|
@xekoukou really interesting, did you have an example of |
|
At the moment, I am reading the mirage packages to implement a simple web
server in the agda programming language.
In general, you can use agda instead of OCaml to write a mirage unikernel.
…On Tue, May 28, 2019 at 7:16 PM Calascibetta Romain < ***@***.***> wrote:
@xekoukou <https://github.com/xekoukou> really interesting, did you have
an example of malfunction + mirage?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#969?email_source=notifications&email_token=AAG2FVGUVEJOQL6KGM33GULPXVLF7A5CNFSM4GWZ7RJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWMUZPY#issuecomment-496585919>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAG2FVGYYDS5CXV2VML7UKLPXVLF7ANCNFSM4GWZ7RJA>
.
|
|
I still slightly fail to understand the high-level view, and connection between tools. afaiu duniverse is something to-be-integrated with mirage? I'd appreciate some remarks about the proposed workflows, both as a developer (where monorepo and no opam is fine) and as a CI system (where installing dependencies via opam is likely beneficial) -- also in respect to david's comments (will the |
|
I've written a recap here https://gist.github.com/TheLortex/f3d92db831b553f6e4eaf2982e3e6427
What's interesting about having a
About the agda on mirageos use-case: these is coq support in dune so hopefully this shouldn't go wrong but I'll keep that in mind. Thanks for the examples link. @hannesm I really hope that it's now clearer for you, but tell me if something is still missing. |
|
@TheLortex thanks for writing this up, this makes the development workflow clearer. now, I'm a bit confused how a (non-interactive) CI flow should work -- is duniverse always required in the new model, or is it possible to only use opam for installing dependencies, and dune for building the unikernel? |
|
The CI flow works the same way: duniverse is needed because packages with C stubs have to be locally compiled with the correct flags. |
|
Discussion still happens in #1020 and concerns about C stubs is highlighted as in this issue. Please read about that when a consensus must be found between all parts. |
|
Status update: the implementation of this feature is now in #1153 which supersedes #1020 and #1024. Here a brief description of the design updates and of the remaining steps. New WorkflowAs in MirageOS 4, a repository can contain multiple $ for c in `list all the config.ml`; do
mirage configure -f $c <params_c>;
done
[..] # generates files, including opam and dune-workspaceNote: this doesn’t work fully yet, all the unikernels have to be configured to use the same target, but that will be fixed before the release. As for MirageOS 3, $ duniverse init
[..] # read all the opam files, resolve depencies and create dune-get
$ duniverse pull
[..] # read dune-get and download all the dependencies in duniverse/Question: do we want a The main new feature of MirageOS 4 is that it is now possible to build all the unikernels in one go: $ dune build @install
[..] # generate all the unikernels in one goOr $ dune exec -- solo5-hvt <path/to/unikernel.hvt>
[..] # run the unikernel in `path/to`Question: do we want to re-introduce a mirage buildIn MirageOS 3,
Question: do we want to deprecate make dependsIn MirageOS 3,
pkg-config and ocamlfind predicatesIn MirageOS 3, the compilation of C bindings were using pkg-config and an extra predicate in ocamlfind META files to replace, at link time, the default C archives by the one needed by the target. This is not the case in MirageOS 4 anymore.
Porting the runtime is a bit more tricky, as it shouldn’t be re-using the workspace CFLAGS that it is defining. To simplify things a bit, samoht/ocaml-solo5 is wrapping the build of solo5 and freestanding into one package per target. This allow all these packages to be co-installable. Current Status#1153 now seems to work for
|
|
Discussion continues in #1195 |
I used several unikernel samples to explore what are the problems blocking the Dune transition. I've taken them from
mirage-skeleton. After identifying problems and workarounds, I managed to build entirely with Dune + ld/pkg-config the following unikernels with unix, xen and hvt target: DNS, http_fetch, echo_server. I'm aware this does not cover everything but it's a good start.Used features of OCamlbuild that aren't implemented in Dune:
-dontlink <pkg>used forunix,threads,strandnum. This removes these packages from the final linking step, thus allowing to link on non-Unix targets.-output-objin conjunction with..._linkoptsin META files to link C stubs and custom OCaml runtime.Workaround to each problem
bigarraywhich includes undefined stubs on xen/freestanding.packageas the default implementation and make it depend on the virtual librarypackage-virtual. This way the update is transparent for users of the package and the ones that want to use the virtual library feature just have to change theirpackagedependency into apackage-virtualdependency.-output-complete-objto build object files. This also links C stubs and runtime. Therefore we don't have to care about C stubs anywore as they are automatically linked along with used libraries. No need for..._linkoptsflags in META. For the custom runtime,ocamlopthas a-runtime-variantoption to choose a custom runtime library to link.Packages to update
lib...asmrun.ato ocaml runtime directory so that ocaml finds the runtime variants. I've already modified these packages locally to make my experiments so it's not that hard to make these changes.Issues and comments
How do you feel about this ? How should we proceed to update packages ? Am I missing something ? Feel free to comment !
The text was updated successfully, but these errors were encountered: