Xen Hypervisor Control

[garethodickin]

Developer Contact Info:

sookyung.ahn@boeing.com

Internal MR link

link to mr (TBD)

Licensing

OSS Approved for Upstream

Feature request summary

* Symbols (4): xc_interface_open, xc_interface_close, xc_sched_arinc653_schedule_get, xc_sched_arinc653_schedule_set
* Status: Unikraft does not provide Dom0 management functionality
Note: These are used for multi-module scheduling from a supervisor partition. May require a different architecture on
    Unikraft (direct hypercall, stub, or removal from DomU builds)
* Effort: Architecture decision needed Xen Hypervisor Control

Describe alternatives

* The `libxenctrl`, which is providing necessary symbols, is not usable from Unikraft domains.
* Create new lib for direct hypercall invocation (`HYPERVISOR_sysctl`) from the guest domain to perform ARINC 653 schedule get/set
* `xc_interface_open` and `xc_interface_close` symbols are unnecessary and removed

Related architectures

arm64

Related platforms

Xen and Qemu

Additional context

* `HYPERVISOR_sysctl` is a hypercall for privileged domain command; these symbols should be excluded from unprivileged DomU builds

[michpappas]

Can you provide more context? Is the idea to use Unikraft as Dom0? My understanding was that the use-case was Dom0-less.

From the architecture's point-of-view: There is an ongoing effort to re-arch the platform layer, starting from 0.20.0. This will continue on 0.21.0 and will define the scope of platform-specific functionality that does not belong to platform drivers - i.e. the scope where libxenctrl-like functionality belongs. To unblock this work, if libxenctl functionality is required indeed, what I would propose is to initially implement Dom0 controls within plat/xen, gated by a Kconfig option (e.g. CONFIG_XEN_CTRL or similar).

[garethodickin]

You're correct, the use case is Dom0less, but with a Unikraft unikernel partition configured as a control domain, domC, with privileged hypercalls managing N unprivileged guest partitions.
I believe something like this:

                ┌───────────────────────────────────────────────────────┐
                │                   Xen Hypervisor                      │
                │               (EL2, manages all domains)              │
                └───────────────────────────────────────────────────────┘
                          ▲                               ▲
                          │                               │
                    Control path                       Control path 
               (privileged hypercalls)           (unprivileged hypercalls)
                          │                               │
                          │                               │
            ┌─────────────┴─────────────┐   ┌─────────────┴─────────────┐
            │         domC (priv)       │   │         domU (unpriv)     │
            │ "Control Domain" unikernel│   │  Application unikernel    │
            ├───────────────────────────┤   ├───────────────────────────┤
            │ **Privileged Hypercalls** │   │ **Safe Hypercalls**       │
            ├───────────────────────────┤   ├───────────────────────────┤
            │ **Shared Memory Regions** │   │ **Shared Memory Regions** │
            │                        ◄──│── |──►                        │
            ├───────────────────────────┤   ├───────────────────────────┤
            │ **vCPU Shared Info Page** │   │ **vCPU Shared Info Page** │
            └───────────────────────────┘   └───────────────────────────┘
                            ▲                         ▲
                            │ Event Channel Signals   │
                            └───────────────┬─────────┘
                                            │
                                            ▼
                              ┌───────────────────────┐
                              │Event Channel Mechanism│
                              │ (in Xen hypervisor)   │
                              └───────────────────────┘

[sookyung-ahn]

I was initially considering lib/ukxenctrl with gating Kconfig option, CONFIG_LIBUKXENCTRL, but your suggestion to keep it within plat/xen/ makes more sense given the re-arch context. Will go with that approach.

[matthew-l-weber]

We should consider the generic hypercalls today that could be exposed for a domC to use. Do we have an example of those we should use first? Then our additional call (that needs the Xen upstream part also fully published) could be enabled with a CONFIG_ARINC653 conditional on the Unikraft side, similar to the current Xen ARINC653 scheduler being selectively enabled plus hypercall on the Xen side.

[michpappas]

There is a topic I wanted to discuss on the last FuSa call, particilarly whether the approach implementd in #1881 is after all the right direction, as it essentially sets the base of what can ultimately grow into a parallel implementation of the upstream library.

Looking at Xen 4.20.0, libxenctrl consists of different parts:

USELIBS_ctrl := toollog call evtchn gnttab foreignmemory devicemodel

Most of these involve an OS-specific layer and there is support for minios (ctrl/xc_mini_os.c, call/minios.c, eventchn/minios.c, gnttab/minios.c, foreignmemory/minios.c).

Since Unikraft's plat/xen/ was originally based on minios it should be possible to build against the upstream implementation.
So what I suggest is to create an external library similar to lib-lwip that imports libxenctrl's upstream sources, adds boilerplate to build as a unikraft library, plus some small glue code / patches.

I had a look on what it would take to support the ARINC653 hypercalls and the changes seeem small:

On Unikraft's side you'd only need to export the declaration of HYPERVISOR_multicall, stub out map_frames_ex(), and add the definitions of L1_PROT / L1_PROT_RO.

On the library's side you'd need to add include/mini-os/ with headers that include stdlib / unikraft headers, glue / stubs for toollog calls, and guard memory helpers for cases like ARINC653 that don't require them, and a few tiny changes here and there.
Disclaimer: I quickly put this together to get an rough idea of how much work is required, and only tested if it compiles.

Misc topics on pursing this direction:

1. Scope: It's not clear to us what subset of libxenctrl is in the scope of the project.
2. License: libxenctrl's sources are LGPL-2.1. That allows static linkage under certain prerequisites that you need to be aware of.
3. Maintainership: If interested and willing to commit, you can own maintainership of that library - we can help a bit at first to establish good practices. Initial support can be limited to the subset required the current project, and later can grow as more use-cases come.