[RFC] Preflight-check / node feature discovery

[c3d]

Is your feature request related to a problem? Please describe.

At the moment, a tool like kata-deploy assumes that it is possible to deploy and that thing will run. In some cases, it ensures that things will run for example by installing its own version(s) of qemu, firecracker and cloud hypervisor.

It would be nice if there was instead a way to run a preflight-check to test that the installation will be successful, and automatically detect whatever software exists on the nodes (or can easily be installed through other means, e.g. for distros). This could be integrated in the kata-deploy script, in the same way kubeadm init does some preflight checks.

This is not necessarily restricted to kata-deploy, the same idea is being explored for a kata operator as well.

Among tests that could be implemented that would help today:

• Verifying that virtualisation is enabled on the target nodes
• Verifying that the relevant hypervisor software is present and functional (some of this already exists in kata-deploy). For example, if there is a distro-supplied qemu, should we use that instead of our own build?
• Does it really make sense to install all hypervisor? Does that match actual usage?
• Making sure that nodes are consistent. For example, should we report if some nodes have inconsistent host kernel version lacking support for 9p or virtiofs or some other virtio?
• Detecting if we run under crio or containerd and if their configuration / version works for us

Tests that could be implemented in the future:

• Checking if some specific hardware exists on the target nodes (node feature discovery, but with a focus on what Kata needs to know), e.g. GPUs, vGPUs, SR-IOV devices, other accelerators. This could impact the configuration of the guest kernel we deploy.
• Checking advanced / future CPU features, like memory encryption
• Verifying network and storage connectivity. This is more forward looking, but in the context of recent discussions regarding the mapping of block devices, it becomes relevant to know if a given block device is for example network-attached on the host, since in that case it might be better to network-attach it to the guest.

The reason for listing the examples above is because this suggests a relatively extensible solution, one where the support for the various use cases above would be "pluggable" through some extensible mechanism.

Describe the solution you'd like

The description below is not necessarily what I would like, more like a possible way to achieve some of the obecjtives above. This is not necessarily the only one, and it may be too early to go in that direction.

• Split the kata-deploy container into four sub-containers: one containing the installation scripts, and three more containing the artefacts we can deploy (i.e. qemu, firecracker and clh). Benefit at this stage: being able to run kata-deploy to only deploy qemu if that's the only component you plan to run. If we do that, we might want to get rid of the separate virtiofs variant of qemu, or create a fifth container with that.
• Add a kata-preflight script to the image, that would run a number of tests on the target nodes, and collect the data to be used for example as extra configuration parameters in the installed configuration files.
• The kata-preflight could itself delegate part of the work to a kata-preflight.d directory, part of which could be read from the host, the intent being that the host owner can deploy (through other means) a way to report specific information about their nodes that they care about (e.g. TDX attestation provider URL, that kind of thing).
• The kata-preflight would run the scripts in turn, collect the data e.g. in JSON format, and convert some of that into something that would be inserted using a template mechanism into the configuration files. For instance, it could detect "no 9p support" and then force the shared configuration to virtiofs (and not bother installing the non-virtiofs qemu).

Describe alternatives you've considered

Alternatives would include:

• An openshift operator doing the work above. The openshift operator now exists, and doing some level of preflight is being discussed and/or worked on.
• A kubernetes operator, i.e. a go implementation of the ideas above. This would be a significant rewrite and the integration with kata-deploy might be more complicated.
• Interfacing with existing node feature discovery interfaces. At the moment, it looks like a number of node features being discussed here would only be relevant to kata, so getting traction outside of the kata project might be difficult. Also, that would make the connexion with kata configuration more difficult.

** Related work **
This is distantly related to efforts like #1709 (qemu feature discovery) or #1708 (being able to install without docker nor k8s), and was discussed during the vPTG on April 19, 2021.

This is also distantly related to features like "Node feature discovery" and may need to interact with them.

[egernst]

This covers a lot - thanks for writing this out @c3d. I definitely think it makes sense to expand what is carried out by kata-check. That'll enable folks to do a preflight check before installing Kata, whether it be in a cluster or on a single node. After that we can examine whether it makes sense to do that in a daemonset across a cluster.

My initial reactions when thinking through this is the trade off between complexity/maintainability versus end user benefit. Who do you think the user would be here?

I can think of 2 general types of users, though I'm sure there's plenty in between.:

1. Someone who wants to quickly kick the tires. This is who I had in mind when we started kata-deploy.
2. Someone who plans to use Kata to solve their problem (security isolation? compliance? perf isolation?) for their existing environment

For RH users, do you think it'd be covered by above, or by a different kind?

For (1) I think "good" error messages and just using the static binaries provided is enough.
For (2), I think that having auto-configurability is probably not desirable -- we should have a sane default that works, and the admin/user should be explicit about what they want setup (and knowledge of cluster). In a heterogeneous cluster, I'd be concerned about running a DaemonSet that automatically configures the runtime differently on each host. At the very least, if I were using it, I would want to be very explicit about this, and make changes to kata-deploy to also label the nodes appropriately per configuration, and register runtimeclasses with similar nodeSelects, etc. Ie, this is very specific to how I would manage my infra.

Basically, having a preflight check is interesting; I think leveraging existing tool (kata-check) would make most sense. I don't like the idea of "auto-configuration" based on detections - this is getting too specific to a users' infra and goals in using Kata imo, and we should make sure users are explicit about their usage.

Curious to hear what other folks think, and I'l keep thinking through this on my end, too!