Runtime merge proposal

[sameo]

Problem Statement

The Kata Containers project is a merge of 2 hardware virtualization based runtimes: Intel’s Clear Containers and Hyper’s runV. As of Feb 13th, 2018, the two runtimes live alongside under the Kata Containers runtime github repository. Having two codebases is confusing the community and impeding potential contributors to really commit to the project.

Although each runtime has its strengths and gaps, the next step for the Kata Containers project should be about combining the two runtimes into a single code base, and merging each runtime feature set into a larger one.

To reach that goal, these two approaches are proposed:

1. Have the community pick one runtime and then work on filling the feature gaps.
2. Find a technical way to pick the best components from each runtime and merge them into a single code base.

During the Feb 5th, 2018 Kata Containers Architecture meeting, the community decided to initially go with the first solution, i.e. picking one runtime over the other.
We believe this solution is a little too aggressive and may underestimate the engineering costs of filling the technical gaps of either runtime.

Goals

The goal of this proposal is to show that the second approach is viable and can be implemented. We think by mixing the best of the two runtime components we can accelerate the runtime merging process and reach the Kata Containers 1.0 release sooner.
This proposal aims at:

• Building a flexible and extensible runtime based on a clean core framework
• Providing a smooth upgrade path for both runV and Clear Containers users, by keeping all features from both runtimes early in the merge process
• Accelerating the runtime merge process
• Providing a base for efficient CRI implementations, for both OCI based CRI shims and native ones

High Level Architecture

By picking the virtcontainers framework that already implements the entire Clear Containers feature set, together with the runV hypervisor drivers, VM factory library and runtime API, we would combine the 2 runtimes features sets. Building the Kata Containers runtime on this combination of components would thus fulfill the Kata Containers runtime requirements.

The Kata Containers runtime would then be based on the following components:

1. The virtcontainers library would be the core Kata Containers runtime framework.
2. In order to have a hypervisor and architecture agnostic design, the runV hypervisor drivers for Xen, libvirt and kvm-tool would be added to the virtcontainers library.
3. The virtcontainers API would be improved and extended in order to support the current runV and Frakti specific requirements.
4. runV's virtual machine factory.

[sameo]

cc @sboeuf @jodh-intel @egernst @gnawux @laijs @bergwolf @jessfraz @WeiZhang555 @tallclair

[gnawux]

Thank you for submitting the proposal @sameo :)

[gnawux]

Add some basic goals of the merging proposal:

• A well organized and extensible architecture;
• A high efficient workflow, especially for OCI and CRI;
• Smooth upgrade for existing customers of both runV and clear-containers;
• Keeps the features from both runV and clear-containers;
• Accelerate the runtime merging process.

[sameo]

@gnawux Done, please let me know if that looks fine to you.

[jodh-intel]

Thanks @sameo!

lgtm

[sameo]

cc @jon

[jessfraz]

This seems sane to me!

[tallclair]

and the runV runtime API

I didn't see that listed below, or is that the same thing as runV's virtual machine factory.?

virtcontainers API would be improved and extended

Would it be possible to list the specific places it falls short here? Or would that require a more involved analysis?

By picking the virtcontainers framework

I'd be interested in seeing a bit more justification for this, even if it's just one sentence. What's the alternative, and why is this preferred?

[sameo]

@tallclair

By picking the virtcontainers framework

I'd be interested in seeing a bit more justification for this, even if it's just one sentence. What's the alternative, and why is this preferred?

virtcontainers is where the entire Clear Containers logic lives. The Clear Containers runtime (cc-runtime) is essentially a translation layer from OCI to the virtcontainers API. As such, by picking the virtcontainers framework, we'd get the entire Clear Containers feature set. And then by adding the VM factory library (for the VM templating feature) and the missing hypervisors support from runV, we'd be closing the feature gaps and meeting the requirements listed in #31.

Do you feel that's clearer?

[sameo]

virtcontainers API would be improved and extended

Would it be possible to list the specific places it falls short here? Or would that require a more involved analysis?

It needs some further analysis. We need to compare the current virtcontainers API with PR #32 to fully where the gaps are, if any.

[tallclair]

Thanks for clarifying. LGTM.

[bergwolf]

What about multi-architecture support? The above summary seems to only mention multi-hypervisor support. The runV hypervisor driver is both hypervisor-agonistic and architecture-agonistic. It has support for ARM64, AMD64, PPC64 and s390x. Are we going to only keep the hypervisor-agonistic part of it?

[bergwolf]

and the runV runtime API
I didn't see that listed below, or is that the same thing as runV's virtual machine factory.?

@tallclair Not just the VM factory. The entire set of API is being discussed here #32

[WeiZhang555]

This LGTM, I like this idea.

One thing, currently virtcontainers resides under https://github.com/containers organization, is it possible to move it back under https://github.com/kata-containers ? Or we can't manage codes and it also tear kata-containers down to too many pieces. @sameo

[bergwolf]

@sameo one nit about the architecture graph -- the agent/shim/proxy are outside of virtcontainers and shim/proxy are not used by frakti.

[sameo]

@bergwolf

Are we going to only keep the hypervisor-agonistic part of it?

Certainly not. It has to be architecture agnostic as well, and as an example we recently merged ARM support for QEMU into the virtcontainers. I'll add a mention to it to the proposal, thanks.

[sameo]

@WeiZhang555

One thing, currently virtcontainers resides under https://github.com/containers organization, is it possible to move it back under https://github.com/kata-containers ? Or we can't manage codes and it also tear kata-containers down to too many pieces.

We initially put virtcontainers under the github.com/containers organization to make it very clear that it's neither vendor nor Clear Container specific. If we decide to go and base the Kata Containers runtime off virtcontainers, it would make perfect sense to move it under github.com/kata-containers/runtime/ as a runtime package. Or maybe as a separate repo, but that would be more complex.

I will send a PR for that purpose tomorrow or early next week, I first need to know what was the output of yesterday's discussion on this proposal.
I'd like to move virtcontainers

[sameo]

@bergwolf

the agent/shim/proxy are outside of virtcontainers and shim/proxy are not used by frakti.

Right, what I meant with this diagram is to show that virtcontainers provide support for managing different kind of proxies, shims and agents. Not that the proxies and shims implementations are done inside virtcontainers. Not sure how to make that clearer with such high level diagram though...

[bergwolf]

@sameo

Not sure how to make that clearer with such high level diagram though...

How about renaming virtcontainers to kata-runtime, and kata-runtime in the graph to kata-cli? Then we can link shim/proxy under kata-cli and agent under kata-runtime. Current separation of kata-runtime and virtcontainers makes it unclear because virtcontainers essentially is part of kata-runtime.

[bergwolf]

@sameo I've drawn a sample architecture graph based on the above suggestion. Please feel free to modify or use it directly. The components in orange are kata components.

[mcastelino]

@sameo @bergwolf can we add a bit more detail on the features provided in the VM templating (VM Factory) component. The rest of the items in the architecture are mostly self explanatory.

[egernst]

Since we have the initial requirements pr merged and virtcontainers is now merged, can this issue be closed?

I'll open a new issue to detail vm templating.