Xen PVH: Third time lucky #1159

mato · 2020-06-04T18:35:37Z

After discussing the state of the Xen platform stack with @mirage/core last month, we decided to explore the option of replacing Mini-OS entirely using a hybrid approach with Solo5 providing the low-level "early boot" code and Mirage providing the rest of the stack.

This approach has the advantage of starting from scratch with a minimal and modern code-base, and gets the existing build system infrastructure and freestanding OCaml runtime provided by Solo5 "for free".

I'm happy to report that this is now past PoC stage and the general approach has been shown to work, and I'm opening this issue to track the remaining work in public. This is a replacement for both #998 (Unikraft) and #1007 (update to Mini-OS master).

Overall status: Unikernels from mirage-skeleton build and run, vchan test unikernels from ocaml-vchan likewise. Tested on Xen 4.11, should work on 4.10 or higher.

Qubes OS status: ~~Qubes OS 4.0 ships with Xen 4.8 which we do not support. These changes need to be tested against the upcoming Qubes OS 4.1 release (#1159 (comment)).~~ Support for Qubes OS custom Xen 4.8 version is now part of mirage/mirage-xen#23.

Non-goals: The existing ARM32 support is removed, this is all x86_64 only for now. Only Xen PVH v2 is supported, available from Xen 4.10 onwards.

There is a package universe available in mirage/mirage-dev, based on Mirage 3.x (currently 3.8.1):

opam repo add mirage-dev-3.x+xen-pvh-via-solo5 git+https://github.com/mirage/mirage-dev.git#3.x+xen-pvh-via-solo5

The following components make up this work. Anything below which does not yet have an active PR is liable to get re-based at any time:

Component	OPAM Package	Branch	Issue/PR	Released
Solo5 Xen bindings	solo5-bindings-xen	mato/solo5#xen	Solo5/solo5#468	0.6.7 ✔️
Freestanding runtime	ocaml-freestanding	mato/ocaml-freestanding#xen	mirage/ocaml-solo5#81	0.6.3 ✔️
Core platform bindings	mirage-xen	mato/mirage-xen#xen-pvh-via-solo5	mirage/mirage-xen#23	6.0.0 ocaml/opam-repository#17444 ✔️
Mirage tool	mirage	mato/mirage#3.8+xen-pvh-via-solo5	#1183	3.9.0 ocaml/opam-repository#17485 ✔️

The following dependencies need API updates, no functional changes intended:

Component	OPAM Package	Branch	Issue/PR	Released
Block	mirage-block-xen	mato/mirage-block-xen#xen-pvh-via-solo5	mirage/mirage-block-xen#87	2.0.0 ocaml/opam-repository#17454 ✔️
Bootvar	mirage-bootvar-xen	mato/mirage-bootvar-xen#xen-pvh-via-solo5	mirage/mirage-bootvar-xen#45	0.8.0 ocaml/opam-repository#17455 ✔️
Console	mirage-console-xen, mirage-console-xen-backend	mato/mirage-console#xen-pvh-via-solo5	mirage/mirage-console#85	4.0.0 ocaml/opam-repository#17456 ✔️
Crypto	mirage-crypto-pk	OPAM metadata change only		0.8.6 ocaml/opam-repository#17459 ✔️
~~Io_page~~	~~io-page-xen~~	~~mato/io-page#xen-pvh-via-solo5~~	(no longer required)
Network	mirage-net-xen, netchannel	mato/mirage-net-xen#xen-pvh-via-solo5	mirage/mirage-net-xen#99	2.0.0 ocaml/opam-repository#17457 ✔️
Qubes	mirage-qubes	OPAM metadata change only		0.9.0 ocaml/opam-repository#17471 ✔️
Vchan	vchan-xen	mato/ocaml-vchan#xen-pvh-via-solo5	mirage/ocaml-vchan#136	6.0.0 ocaml/opam-repository#17458 ✔️

Rough TODO list

PoC works, Solo5 Xen bindings complete, Mirage "hello, world" works
Functionally complete (builds mirage-skeleton, ocaml-vchan tests)
Clean up interfaces in mirage-xen (naming of C stubs, ~~OS module~~ fold xen-evtchn into mirage-xen)
All components above have PRs and review is in progress
Wider testing of the new Xen stack, especially from Qubes OS users (see Upgrade to Mirage 6 for solo5 PVH support qubes-mirage-firewall#116)
Merge PRs, cut releases and PR to opam-repository, followed by a Mirage 3.x release (see "Released" column in above tables for progress)
Forward-port to Mirage 4 (see Xen pvh via solo5 for master #1193)

The text was updated successfully, but these errors were encountered:

avsm · 2020-06-05T10:03:36Z

Really nice work! Had a quick look at the Bootvar and Io_page API changes, and both look fine. It's lovely to see the MiniOS internals disappear from the bindings.

mato · 2020-07-03T15:19:50Z

Updated to reflect the current status, WIP branches and added a rough TODO list. This is now "almost" functionally complete barring some rough edges which I hope to have sorted soon, after which I will put out a call for wider testing while working on getting the WIP branches into shape for PR and actual review.

hannesm · 2020-07-09T15:32:48Z

I tried your branches / opam repo on my Qubes laptop. It does not work out of the box unfortunately (the start_info is version 0 here, not >= 1 -- Xen version: 4.8.5-14.fc25). The output of a unikernel (I added a log for the command line and moved the assert(si->version >= 1) some lines below:

Solo5: Xen console: port 0x2, ring @0x00000000FEFFF000
Solo5 Xen start_info->version 0x0
Solo5 Xen cmdline root=/dev/mapper/dmroot ro nomodeset console=hvc0 rd_NO_PLYMOUTH rd.plymouth.enable=0 plymouth.enable=0 rd
Solo5: ABORT: xen/platform.c:142: Assertion `si->version >= 1' failed

AFAICT, solo5 platform uses the start_info to figure out the memory size as well, which is not present in version 0 (used by QubesOS). Reading a bit of mini-os, it looks like it's possible to use a different path (https://github.com/mirage/mini-os/blob/48d3b31ce47156d92255a2fe98334dd15a42a097/arch/x86/mm.c#L127). Though, I'm not sure what the timeline of Xen updates in QubesOS are (maybe @marmarek has some comments here?) in order to get a newer start_info struct.

A second issue outlined above is that it looks like QubesOS needs the boot parameter filters, since it's passing something along that won't work with cmdliner/functoria/mirage.

The output of dmesg in the host system:
dmesg-hannes.txt

(@mato: Edited to replace 700 lines of dmesg with an attachment)

marmarek · 2020-07-09T15:40:59Z

A second issue outlined above is that it looks like QubesOS needs the boot parameter filters, since it's passing something along that won't work with cmdliner/functoria/mirage.

Already available: qvm-features VMNAME no-default-kernelopts 1
Then, kernelopts property is the only thing that gets passed to the unikernel and you can set it freely to any value.

Though, I'm not sure what the timeline of Xen updates in QubesOS are (maybe @marmarek has some comments here?)

Qubes 4.1 should be out in few months (rc1 probably next month, then 2-3 months until final release). This will have Xen 4.13.
But Qubes 4.0 will be there for a while, so if supporting older Xen too isn't too complex, I think it's worth adding it.

mato · 2020-07-09T15:47:22Z

@marmarek

Qubes 4.1 should be out in few months (rc1 probably next month, then 2-3 months until final release). This will have Xen 4.13.
But Qubes 4.0 will be there for a while, so if supporting older Xen too isn't too complex, I think it's worth adding it.

The goal of this work was to eliminate/replace as much legacy code as possible, with prejudice. Therefore, I wrote it to target the Xen PVHv2 ABIs only. Given that Xen only lists PVHv2 as "fully supported" from 4.10 onwards, I would strongly prefer not to try and support earlier versions.

marmarek · 2020-07-09T15:53:29Z

Fair enough.

hannesm · 2020-07-14T13:01:24Z

hi @marmarek ,

qvm-features VMNAME no-default-kernelopts 1

thanks, that works nicely.

kernelopts

NB this is part of qvm-prefs (not qvm-features).

4.1rc likely next month

great, I'm looking forward to test this work on qubes with a more recent Xen. I'd be fine to test an early beta of 4.1, if there's documentation for how to upgrade from a Qubes 4.0 (or a binary usb stick release so I can reinstall my QubesOS). please let us know when there's such an upgrade path or beta release.

Update mirage-block-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.

Update mirage-console-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.

Update mirage-net-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.

Update vchan-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.

This adapts the `mirage` front-end tool to the new Mirage/Xen platform stack (see mirage#1159). User-visible changes: - `.xe` (Xenserver) filegeneration has been removed, I have no way to test this. - "Unknown" argument filtering in `Bootvar` / `argv` has been removed, this was causing problems and I do not know of any users apart from Qubes OS where this can be solved differently, see comments in mirage#1159. Note that Xen unikernels now include a dummy Solo5 manifest to keep the rest of the build system and Solo5 layers happy; this does not contain any devices as they are not required and Xen device naming does not follow Solo5 requirements.

mato · 2020-07-24T16:39:41Z

All components now have PRs ready for review; I've updated the description here with the current status.

mato · 2020-08-13T15:52:59Z

See #1184 for discussion / work in progress on supporting the Xen version shipped by the existing Qubes OS 4 release.

This adapts the `mirage` front-end tool to the new Mirage/Xen platform stack (see mirage#1159). User-visible changes: - `.xe` (Xenserver) filegeneration has been removed, I have no way to test this. - "Unknown" argument filtering in `Bootvar` / `argv` has been removed, this was causing problems and I do not know of any users apart from Qubes OS where this can be solved differently, see comments in mirage#1159. Note that Xen unikernels now include a dummy Solo5 manifest to keep the rest of the build system and Solo5 layers happy; this does not contain any devices as they are not required and Xen device naming does not follow Solo5 requirements.

mato · 2020-10-07T16:17:02Z

Status update: Qubes OS 4.0 support is now done (Solo5/solo5#480 and added to mirage/mirage-xen#23). Most of the discussion is in mirage/qubes-mirage-firewall#116. Further fixes to importing grant mappings from other Xen domains have landed in mirage/mirage-xen#23.

I have updated the package universe (see description) to Mirage 3.8.1 and to include the above; you should be able to build Mirage/Xen unikernels for both Xen 4.10+ and Qubes OS 4+ from it with no problems.

This adapts the `mirage` front-end tool to the new Mirage/Xen platform stack (see mirage#1159). User-visible changes: - `.xe` (Xenserver) filegeneration has been removed, I have no way to test this. - "Unknown" argument filtering in `Bootvar` / `argv` has been removed, this was causing problems and I do not know of any users apart from Qubes OS where this can be solved differently, see comments in mirage#1159. Note that Xen unikernels now include a dummy Solo5 manifest to keep the rest of the build system and Solo5 layers happy; this does not contain any devices as they are not required and Xen device naming does not follow Solo5 requirements.

* [Mirage 3.x] Port to new Mirage/Xen platform stack This adapts the `mirage` front-end tool to the new Mirage/Xen platform stack (see #1159). User-visible changes: - `.xe` (Xenserver) filegeneration has been removed, I have no way to test this. - "Unknown" argument filtering in `Bootvar` / `argv` has been removed, this was causing problems and I do not know of any users apart from Qubes OS where this can be solved differently, see comments in #1159. Note that Xen unikernels now include a dummy Solo5 manifest to keep the rest of the build system and Solo5 layers happy; this does not contain any devices as they are not required and Xen device naming does not follow Solo5 requirements. Other changes: - Checksum stubs are provided by mirage-xen, tcpip.xen is not needed to be linked anymore. - mirage-{net,bootvar,block,console}-xen, mirage-qubes{-ipv4}: adapt bounds for the upcoming release - opam: provide a dev-repo field to silence warnings Co-authored-by: Hannes Mehnert <hannes@mehnert.org>

hannesm · 2020-10-24T11:35:06Z

Update: the dependencies have been merged into opam-repository, and mirage 3.9.0 is tagged and PRed to opam-repository. PR #1193 is a forward-port of the changes to master. We plan to announce the release on mailing lists etc. once mirage 3.9.0 is merged into opam-repository.

mato · 2020-10-27T14:13:36Z

All done here!

This was referenced Jun 4, 2020

Xen bindings Solo5/solo5#468

Merged

MirageOS-xen goes PVH (using an updated mini-os) #1007

Closed

mato pinned this issue Jul 3, 2020

mato mentioned this issue Jul 3, 2020

Support for Xen domU kernel on aarch64 #1057

Closed

hannesm mentioned this issue Jul 3, 2020

MirageOS 4.0 draft release roadmap #994

Closed

37 tasks

This was referenced Jul 3, 2020

Add native ARM64/Xen support #463

Closed

Add HVM booting support to MiniOS #313

Closed

This was referenced Jul 21, 2020

Add support for Solo5/xen bindings, posix_memalign() mirage/ocaml-solo5#81

Merged

Solo5 v0.6.6 ocaml/opam-repository#16850

Merged

ocaml-freestanding 0.6.1 ocaml/opam-repository#16852

Closed

New Xen core platform stack mirage/mirage-xen#23

Merged

mato added a commit to mato/mirage-block-xen that referenced this issue Jul 24, 2020

Update to new Xen core platform stack

c6787e3

Update mirage-block-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.

mato added a commit to mato/mirage-console that referenced this issue Jul 24, 2020

Update to new Xen core platform stack

6386bbc

Update mirage-console-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.

mato added a commit to mato/mirage-net-xen that referenced this issue Jul 24, 2020

Update to new Xen core platform stack

041050a

Update mirage-net-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.

mato added a commit to mato/ocaml-vchan that referenced this issue Jul 24, 2020

Update to new Xen core platform stack

70abe3b

Update vchan-xen to the interface changes in the new Xen core platform stack. Part of mirage/mirage#1159, depends on mirage/mirage-xen#23.

This was referenced Jul 24, 2020

Update to new Xen core platform stack mirage/mirage-block-xen#87

Merged

Update to new Xen core platform stack mirage/mirage-bootvar-xen#45

Merged

This was referenced Jul 24, 2020

Update to new Xen core platform stack mirage/mirage-console#85

Merged

Update to new Xen core platform stack mirage/mirage-net-xen#99

Merged

Update to new Xen core platform stack mirage/ocaml-vchan#136

Merged

mato mentioned this issue Jul 24, 2020

[Mirage 3.x] Port to new Mirage/Xen platform stack #1183

Merged

mato mentioned this issue Aug 13, 2020

Support for Qubes OS 4 in new PVH-only Xen stack #1184

Closed

mato mentioned this issue Oct 7, 2020

Xen: various improvements Solo5/solo5#480

Merged

mato mentioned this issue Oct 7, 2020

Upgrade to Mirage 6 for solo5 PVH support mirage/qubes-mirage-firewall#116

Merged

This was referenced Oct 18, 2020

opam admin add-constraint 'mirage-xen<6.0.0', and further conflicts ocaml/opam-repository#17430

Merged

future of mirage-platform mirage/mirage-platform#199

Closed

hannesm mentioned this issue Oct 25, 2020

README: use kernelopts='' instead of None mirage/qubes-mirage-firewall#117

Merged

mato closed this as completed Oct 27, 2020

mato unpinned this issue Oct 27, 2020

hannesm mentioned this issue Oct 28, 2020

HVM support mirage/qubes-mirage-firewall#29

Closed

hannesm mentioned this issue Nov 14, 2020

Mirage 4.0, dune and cross-compilation #1195

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Xen PVH: Third time lucky #1159

Xen PVH: Third time lucky #1159

mato commented Jun 4, 2020 •

edited by hannesm

Loading

avsm commented Jun 5, 2020

mato commented Jul 3, 2020

hannesm commented Jul 9, 2020 •

edited by mato

Loading

marmarek commented Jul 9, 2020

mato commented Jul 9, 2020

marmarek commented Jul 9, 2020

hannesm commented Jul 14, 2020

mato commented Jul 24, 2020

mato commented Aug 13, 2020

mato commented Oct 7, 2020

hannesm commented Oct 24, 2020

mato commented Oct 27, 2020

Xen PVH: Third time lucky #1159

Xen PVH: Third time lucky #1159

Comments

mato commented Jun 4, 2020 • edited by hannesm Loading

Rough TODO list

avsm commented Jun 5, 2020

mato commented Jul 3, 2020

hannesm commented Jul 9, 2020 • edited by mato Loading

marmarek commented Jul 9, 2020

mato commented Jul 9, 2020

marmarek commented Jul 9, 2020

hannesm commented Jul 14, 2020

mato commented Jul 24, 2020

mato commented Aug 13, 2020

mato commented Oct 7, 2020

hannesm commented Oct 24, 2020

mato commented Oct 27, 2020

mato commented Jun 4, 2020 •

edited by hannesm

Loading

hannesm commented Jul 9, 2020 •

edited by mato

Loading