re-add OCI CLI commands for podman (Note: docker is now supported with docker-23.0)

[jodh-intel]

We should consider re-adding the OCI commands (removed on #332, #363) so we can support docker and podman with Kata 2.0 (since those container manager do not support shim v2 (yet).

[c3d]

For the record (to make search easier), here is what you get when you try:

#  podman run -it --rm  --runtime /usr/local/kata/usr/local/bin/containerd-shim-kata-v2  fedora bash                                 io.containerd.kata.v2: shim namespace cannot be empty
ERRO[0005] Error removing container 3a073fca6248bd92d5e9917d49df7220fc5113d64664901596b5621777049f95 from runtime after creation failed 
Error: flag provided but not defined: -systemd-cgroup
Usage of /usr/local/kata/usr/local/bin/containerd-shim-kata-v2:
  -address string
    	grpc address back to main containerd
  -bundle string
    	path to the bundle if not workdir
  -debug
    	enable debug output in logs
  -id string
    	id of the task
  -namespace string
    	namespace that owns the shim
  -publish-binary string
    	path to publish binary (used for publishing events) (default "containerd")
  -socket string
    	abstract socket path to serve: OCI runtime error

[c3d]

See also #1133 (comment)

[zer0def]

Just for the record, you can still use Kata v1 with current latest LTS kernel (at the time of writing, 5.10.37, or at least that's what I just tested with) to have an OCI/shimv1 interface for Docker to talk through, since their v2 implementation leaves a lot to expect.

While not a recommended or maintained solution, you can at least stretch it out a bit until this issue is resolved.

[fidencio]

Just for the record, you can still use Kata v1 with current latest LTS kernel (at the time of writing, 5.10.37, or at least that's what I just tested with) to have an OCI/shimv1 interface for Docker to talk through, since their v2 implementation leaves a lot to expect.

While not a recommended or maintained solution, you can at least stretch it out a bit until this issue is resolved.

You can, it's just not maintained, out-of-support, and highly discouraged to do so.

[c3d]

The opposite side of this for podman is containers/podman#8579

[adrecord]

Is this still being worked on? Thanks

[c3d]

@adrecord I am restarting an effort in that direction. See following comments.

[c3d]

There are currently four approaches that have been or are being considered:

1. An "OCI multiplexer" that would act as a gateway between standard OCI command line and the shimv2 interface.
2. A shared library written in C that podman could invoke in the same way they invoke libkrun.
3. Adding support for the command-line to runtime-rs
4. Adding support to the command-line back to the runtime.

[c3d]

Approaches being considered

OCI multiplexer

The OCI multiplexer approach is intended to be totally independent of either podman or Kata. It takes OCI command-line options as input, and convers them to the shimv2 interface.

The good news is that there is a crate for parsing OCI comand line options, as well as to connect to the shimv2 interface.

This approach should not need all the complexity of persisting / restoring the state that we used to have in the old days in Kata Runtime, because the state is in the shim. As long as we can connect to the sim, we should be OK.

Shared library

The podman team was successful calling linkrun that way. See discussion on podman for details. One benefit is that there is one less process, and that some of the required infrastructure is already in podman.

On the other hand, calling ttrpc from C means going through something like protobuf-c. Some quick experiments showed that parsing the required proto files leads to an annoyingly long list of -I options (there is no obvious way to interface with go module dependency management), but it is otherwise doable.

Command-line support in runtime-rs

Long-term, it might be nice to use liboci-cli directly and integrate that into runtime-rs, making it an OCI-compliant runtime again. This approach has two drawbacks:

• It might bloat runtime-rs with features that are only useful from something like podman
• It does not apply to the Go version of the runtime, so it's at best for the long-term.

However, the overhead might be relatively small and worth it to avoid having a separate process.

Command-line support in the Go runtime

The related branch has not been maintained or rebased in an eternity (or more). That approach is seen as going in the wrong direction by the architecture committee. It won't work for runtime-rs either. So let's drop it.

Other considerations

For the split host/tenant API, we need "something" that can sequence RPC commands that need both a host and a guest side, notably for networking setup. While ultimately that split and sequencing can only occur on a machine that the tenant owns (e.g. the laptop used to connect to the cluster) for obvious security reasons, in the shorter term, an intermediate wrapper could be used as a development tool to dispatch commands easily between host runtime and agent.

Conclusion

Currently focusing on the OCI multiplexer first approach.

[struanb]

Due to issues like this one, experienced using Docker to launch Kata Containers, we built RunCVM (Run Container VM): an experimental open-source Docker container runtime, for launching standard container workloads in VMs.

Please note that RunCVM is not a direct competitor to Kata: as an experimental runtime, RunCVM cannot offer the same levels of stability and support as Kata. However RunCVM may be suitable for some use cases and is compatible with docker run today (with experimental support for podman run). Like Kata v2, RunCVM is also virtiofs-based for speed. RunCVM has minimal system dependencies: it relies on the Linux KVM module, and can even be installed in a GitHub Codespace.

I'm sharing this info here in case it's helpful to anyone who, like us, needs VM isolation for their container workloads, needs a virtiofs-based solution (ruling out Kata v1) and needs compatibility with docker run (ruling out Kata v2).

[neersighted]

It's worth noting that since Docker 23.0, Kata Containers v2 is natively supported. Could we remove Docker from the title and clarify that this is really only necessary for Podman in the issue body?