[RFC] Direct Attachable CNIs For Kata Containers

[bergwolf]

Background

Kata Containers is an open source container runtime, building lightweight virtual machines that seamlessly plug into the containers ecosystem. It aims to bring the speed of a container and the security of a virtual machine to its users.
As Kata Containers matures, how it interacts with Kubernetes CNI and connects to the outside network, has become increasingly important. The issue covers the current status of the Kata Containers networking model, its pros and cons, and a proposal to further improve it.

This literrally revives kata-containers/runtime#592, with better explaination on why we need it and how it can be implemented.

Status

A classic CNI deployment would result in a networking model like below:

Where a pod sits inside a network namespace, and connects to the outside world via a veth pair. In order to work with this networking model, Kata Containers has implemented a TC based networking model.

Where inside the pod network namespace, a tap device tap0_kata is created and Kata sets up TC mirror rules to copy packages between eth0 and tap0_kata. The eth0 device is a veth pair endpoint and its peer is a veth device attached to the host bridge. So the data flow is like:

As we can see, there are as many as five jumps before a package can reach the guest on the host. The network stack jumps are costly and the architecture needs to be simplified.

Proposal

We can see that all Kata need is a tap device on the host, and it doesn't care how it is created (being it a tuntap, or a ovs tap, or a ipvtap, or a macvtap). So we can create a simple architecture and use tap devices (or similar devices) as the pod network setup entrypoint rather than veth pairs. Something like:

With this architecture, we can remove the need for a host network namespace, and the veth-pair to connect through it. And we don't care how the tap device is created so that CNI plugins can still have different implementation details hidden from us.

A possible control flow for the direct attachable CNIs:

To make it work, there are a few changes to CNI, containerd and Kata:

• Containerd cri runtime handler config option adds disable_cni_netns and the direct attachable network capability of the underlying runtime net_tap_device_capable = true, net_vhost_user_device_capable, net_vfio_device_capable, and net_vfio_user_device_capable as well).
• CNI plugin config adds options like "tapDevice": true, (and "vhostUserDevice": true etc.) in the capabilities field.
• CNI ADD command adds options like "tapDevice": true (and "vhostUserDevice": true etc.) in the runtimeConfig field.
• CNI ADD command result adds a plugin-specific result field like (for vhost-user/vfio/vfio-user only):

  "hostDevicePaths": [
    {
      "type": "vhost-user-net",
      "path": "/some-vhost-user-socket-path"
    },
    {
      "type": "vfio",
      "path": "/some-vfio-device-path"
    }
  ]

• containerd <-> CNI interaction should include direct attachable network requirement and CNI should return required network device info to containerd
• containerd should pass the network device info (e.g., the CNI ADD command result) to kata in the very first task Create API of a pod.
• Kata needs to consume the network device info and setup pod network properly

Example Workflow for CNI+Containerd+Kata in the Tap Device Case

• containerd cri runtime handler has

[plugins.cri.containerd.runtimes.kata]
    runtime_type = "io.containerd.kata.v2"
    disable_cni_netns = true
    net_tap_device_capable = true

• CNI plugin config has

    "capabilities": {
      "tapDevice": true
    }

• When calling the cni plugins, containerd does not create a new netns for the pod, and computes runtimeConfig to include

    "runtimeConfig": {
      "tapDevice": true
    }

• containerd calls CNI plugin with ADD command with empty NetNS argument and with the following PluginArgs argument: (FIXME: how do we handle IP allocation with kubeovn?)

{
    "cniVersion": "1.0.0",
    "name": "containerd-net",
    "type": "ovn",
    "keyA": ["some additional", "plugin specific", "configuration"],
    "ipam": {
      "tenentID": "pod-tenent-id"
    },
    "runtimeConfig": {
      "tapDevice": true
    }
}

• CNI creates a new tap device in the host netns and returns the following result to containerd:

{
    "ips": [
        {
          "address": "10.1.0.5/16",
          "gateway": "10.1.0.1",
          "interface": 1
        }
    ],
    "routes": [
      {
        "dst": "0.0.0.0/0"
      }
    ],
    "interfaces": [
        {
            "name": "tap10",
            "mac": "00:11:22:33:44:55"
        }
    ],
    "dns": {
      "nameservers": [ "10.1.0.1" ]
    }
}

• containerd passes the above CNI ADD result to Kata in the task CreateTaskRequest struct.
• Kata parses the CNI result, passes tap10 to the vmm and sets it up in the guest via provided IP/route/mac/dns information

Related Projects

There are a few CNI projects that are capable of creating tap devices, and thus can benefit from the direct attachable CNIs. Most notably those built upon OVS:

• kube-ovn: https://github.com/kubeovn/kube-ovn
• antrea: https://antrea.io/

/cc @egernst @amshinde @fidencio

[ariel-adam]

Stefano Brivio is working on the following user space solution that could also be relevant for kata: https://passt.top/passt/about/

[sbrivio-rh]

Stefano Brivio is working on the following user space solution that could also be relevant for kata: https://passt.top/passt/about/

A few details about this: the general idea (not ready for consumption or demo, at this stage) would be to add vhost-user ring support to it, and provide full network capabilities (except for custom IP protocols) without requiring special privileges, while bypassing qemu for the data path, and the possibility of having rootless containers (at least as far as network capabilities are involved).

This is relatively independent from the presence of a further network namespace -- maintaining it allows for easier configuration, and also operation of rootless containers similar to what slirp4netns provides, but indeed dropping the separate network namespace on the host should have significant potential to improve packet transfer performance.

[amnoni]

Some comments/questions:

1. By not having a network namespace on the host, isn't the security being compromised?
2. Should'n the solution be CNI agnostic? Or at least try to accommodate current common CNIs (as many as possible)?

[bergwolf]

@amnoni

1. As being discussed in the kube-ovn issue, netns on the host is optional. Kata doesn't need the netns, but it doesn't mean we cannot work with an netns. I would prefer to leave it to the CNI implementation to decide if it wants to create a netns and Kata just works with what it is given.
2. Yes, the solution is CNI agnostic. But we need CNI to be aware of Kata's characteristics and implement optimizations. So some CNI (like kube-ovn) will be better at working with Kata, while others can catch up eventually.

[sbrivio-rh]

Where inside the pod network namespace, a tap device tap0_kata is created and Kata sets up TC mirror rules to copy packages between eth0 and tap0_kata. The eth0 device is a veth pair endpoint and its peer is a veth device attached to the host bridge. So the data flow is like:

As we can see, there are as many as five jumps before a package can reach the guest on the host. The network stack jumps are costly and the architecture needs to be simplified.

I understand this might have some impact on performance, although this part is there for obvious reasons of isolation (in a logical and in a security sense) and compatibility.

I wonder, though, given that the fast path of those network devices interfaces is rather lightweight and mostly zero-copy, what is the actual performance impact? Do you have some data motivating this?

1. As being discussed in the [kube-ovn issue](https://github.com/kubeovn/kube-ovn/issues/837), netns on the host is optional. Kata doesn't need the netns, but it doesn't mean we cannot work with an netns. I would prefer to leave it to the CNI implementation to decide if it wants to create a netns and Kata just works with what it is given.

I think however the security topic is something that should be considered in this sense: the CNI implementation can indeed decide, but is there a way we can provide equivalent security (e.g. via netfilter) without an additional namespace?

2. Yes, the solution is CNI agnostic. But we need CNI to be aware of Kata's characteristics and implement optimizations. So some CNI (like kube-ovn) will be better at working with Kata, while others can catch up eventually.

As far as I understand this is not actually agnostic in the sense that CNIs need to support that, specifically. I wonder, here, given the peculiarities of the networking scenario in Kata, if it would rather make sense to abstract this and fake the network namespace, optionally -- that could really make this option CNI agnostic.

[bergwolf]

I wonder, though, given that the fast path of those network devices interfaces is rather lightweight and mostly zero-copy, what is the actual performance impact? Do you have some data motivating this?

There are two motivations in the original proposal:

1. the removal of the unneeded veth pair and TC mirroring rules, which should result in lower latency (due to fewer network stacks), and cleaner network setup
2. the possibility to support VFIO and vhost-user based NICs, which is not possible with current veth pair based setup

the CNI implementation can indeed decide, but is there a way we can provide equivalent security (e.g. via netfilter) without an additional namespace?

When security is a concern, we can just create a fake netns when CNI doesn't create one, as you suggested in the comments. I don't see a need for equivalent security solutions. Do you have other concern/usecase in mind?

As far as I understand this is not actually agnostic in the sense that CNIs need to support that, specifically

It is CNI agnostic in the sense that the API extension does not bind to any specific CNI implementation and any CNI can support it if it wants to. And ALL exiting CNIs create netns for PODs at the moment. So I would not rush to implementing a fake netns in kata in the current stage.

[sbrivio-rh]

I wonder, though, given that the fast path of those network devices interfaces is rather lightweight and mostly zero-copy, what is the actual performance impact? Do you have some data motivating this?

There are two motivations in the original proposal:

1. the removal of the unneeded veth pair and TC mirroring rules, which should result in lower latency (due to fewer network stacks), and cleaner network setup

This is actually why I was asking: should -- but do you have some data? Note that the network stack isn't traversed multiple times because of that -- indeed there must be overhead, I just wonder if we already have some indications of it.

the CNI implementation can indeed decide, but is there a way we can provide equivalent security (e.g. via netfilter) without an additional namespace?

When security is a concern, we can just create a fake netns when CNI doesn't create one, as you suggested in the comments.

Wait, sorry, my comment was probably not clear then. By "faking a network namespace" I mean pretending there is one, when in fact there's none. In any case, if it's fake, it serves no purpose, so it shouldn't affect security negatively nor positively.

I don't see a need for equivalent security solutions. Do you have other concern/usecase in mind?

Say ARP spoofing. With the current solution implementing a separate network namespace, we have a rather convenient place (tc filter interface between tap and veth) to drop or forward frames based on their Ethernet addresses. If the tap interface moves outside of a separate network namespace, this possibility is gone.

As far as I understand this is not actually agnostic in the sense that CNIs need to support that, specifically

It is CNI agnostic in the sense that the API extension does not bind to any specific CNI implementation and any CNI can support it if it wants to.

If the CNI needs to know about this, then I would still argue that it's not CNI agnostic. :) But yes, I understand your point, it can be implemented by multiple CNIs, it doesn't require a specific CNI to be implemented for it.

And ALL exiting CNIs create netns for PODs at the moment. So I would not rush to implementing a fake netns in kata in the current stage.

I meant almost the opposite: given that CNIs need a separate network namespace, we keep it, but we might optionally not use it (that's what makes it "fake"), and the CNI doesn't even know about it (that's what makes it agnostic).

[oilbeater]

Here is the performance test result between ovs veth and ovs internal port (implemented by tap device). We use ping and iperf3 to test latency and throughput between two pods on the same node. We also tested macvlan as the baseline

Nic Type	Latency(ms)	Throughput (Gbits/second)
Veth	0.042	26
Internal Port	0.035	31.1
Macvlan	0.026	35.1

Note: the extra TC mirror overhead is not tested

[chestack]

AFAIK, containerd instead of CNI created the netns https://github.com/containerd/cri/blob/master/pkg/server/sandbox_run.go#L122

[oilbeater]

After some test with tap device, once the tap device is moved to netns, the ovs can not find the tap device and lost link to it.

[bergwolf]

After some test with tap device, once the tap device is moved to netns, the ovs can not find the tap device and lost link to it.

@oilbeater Then how about we don't move it to the netns, but instead add a new disable_cni_netns = true option to containerd runtime plugin options.

[sbrivio-rh]

By the way, some months ago, I worked on an alternative approach using (passt)[https://passt.top]. This diagrams (only visible on a bright background, open image separately if needed) outlines the approach:

and I prepared a PoC at https://passt.top/passt/tree/contrib/kata-containers. Mere days after finishing that patch, it wouldn't apply anymore before of a rework in kata-runtime.

This has the advantage of being really agnostic to whatever architecture, not needing root, and it should be performance-wise comparable with a veth pair. I'm not active on this at the moment, I just left this behind: https://bugs.passt.top/show_bug.cgi?id=26. If somebody happens to play with this, I'll be more than happy to support them.

[bergwolf]

@sbrivio-rh Thanks for the nice diagrams. Here my main focus is to let Kata integrate with different network devices (tap, vfio, vhost-user etc.) that may be provided by some VM-native CNIs (such as kubeovn). The main change of the proposal is about telling CNI and containerd to provide these network devices (and related nic setup information) when configured properly.

passt looks to be a sound solution to unify vmm network interfaces. It can be complementary to the above direct attachable NIC proposal if it is able to handle tap/vfio/vhost-user devices. Can you elaborate a bit on that part?

[sbrivio-rh]

@bergwolf with passt you don't need a tap device, that's why I'm mentioning it here. You can have whatever interface you want inside the guest, as long as it can connect to a qemu socket back-end.

That won't work for vfio, indeed.

About vhost-user: we're working on adding it as a back-end, for performance reasons. But this is not so relevant in this sense -- the idea is just to abstract away the network interface between guest and host by making it pretty much transparent. The CNI can then do whatever it needs to do while ignoring the fact that there's a virtual machine.

[bergwolf]

@sbrivio-rh We constantly get user requests to have tap/vfio/vhost-user/physical-nic etc. support in Kata Containers (for performance reasons obviously). While we can support it with some additional control path (such as the kata-runtime network subcommand), it is always our goal to solve it within the context of CNI. That's why I'm looking at the issue from time to time and would like to push it forward constantly.

As for passt, IIUC, it replaces the TC+tap solution. But TC+tap also works for all CNI implementations. The CNI does not need to know if it is runc or kata running there. It seems that the advantage of passt is to allow rootless containers(including rootless crio/containerd/kata-runtime), am I understanding it correctly? It seems that passt is solving a different problem than the direct attachable CNI proposal here.

[sbrivio-rh]

As for passt, IIUC, it replaces the TC+tap solution. But TC+tap also works for all CNI implementations. It seems that the advantage of passt is to allow rootless containers(including rootless crio/containerd/kata-runtime), am I understanding it correctly?

Correct, that was the primary, failed (for the moment) goal. However,

The CNI does not need to know if it is runc or kata running there.

...I was thinking that it might also help with this, because it transform the Kata Containers case into something that looks like a host container, and for that (I guess) you already have full CNI support for other cases requested by the user.

But indeed, it wouldn't work with VFIO, so, if that's the current focus, my suggestion doesn't really apply.

In terms of performance, passt might become useful, also in this perspective, the day it gains native vhost-user support, but that's not available at the moment.

[oilbeater]

Then how about we don't move it to the netns, but instead add a new disable_cni_netns = true option to containerd runtime plugin options.

It sounds like the host network mode. As for now, kubelet will not call CNI if the pod runs with hostnetwork set to true. If we skip kubelet and set it from containerd side, the containerd can pass the host netns to CNI, and the CNI can just follow the usual process but at this time it will not move the tap to other ns.

[justxuewei]

Hi community,

I would like to give a feasible idea for this issue. As discussed in this thread and #4914, the tap devices in the host netns and vhost-user sockets should be attached to the hypervisor as well. Setting up dummy devices in the netns as the Kata containers 2.0 does is not a good idea. We use a file, instead of dummy devices, to exchange the network information because of flexibility and expandability.

The first question: Where are those network configs stored?

The name of netns identifies the network configs of a pod. Assumed that we have a netns, named cni-725a6170-996e-63df-2b80-4e5d75fba6e2, and the default basepath is /var/lib/cni/netconf, which it could be configurable. Eventually, Kata containers will load the network configs at /var/lib/cni/netconf/cni-725a6170-996e-63df-2b80-4e5d75fba6e2.

The second question: What is the format of the network configs?

There are two types at least, vhost-user-socket and tap in the host netns, to be supported. The content of the file is an array in the format of json as shown in the following.

[{
	"name": "example",
	// the possible values are "vhost-user" or "host-tap"
	"type": "vhost-user",
	// !!todo: to be determined
	"dev_conf": {
		"path": "/var/run/openvswitch/vhost_sockets/sock"
	},
	"network_info": {
		"interface": {
			"ip_addrs": [{
				"family": "v4",
				"address": "192.168.1.1",
				"mask": "16"
			}],
			"device": "device_name",
			"hardware_addr": "xx:xx:xx:xx:xx",
			"mtu": 1500,
			"raw_flags": "0x11"
		},
		"routers": [{
			"dest": "172.18.0.0/16",
			"src": "172.18.0.1",
			"gw": "172.18.31.1",
			"scope": 1
		}],
		"neighbors": [{
			"to_ip_addr": {
				"family": "v4",
				"address": "192.168.1.1",
				"mask": "16"
			},
			"state": 0,
			"flags": 0,
			"hardware_addr": "xx:xx:xx:xx:xx"
		}]
	}
}]

The last question: How do the Kata containers collaborate with the CNI plugins to set up the network?

The CNI plugins should set up net devices and save the network configs into the path we mentioned above. Before scanning the network devices in the netns, the Kata containers will load the json config and set up corresponding endpoints if the file exists.

The process of setting up the network looks like this:

[liubin]

@justxuewei

We may use RuntimeConf.ContainerID instead of netns, so that we can support network without creating a new netns.

// A RuntimeConf holds the arguments to one invocation of a CNI plugin
// excepting the network configuration, with the nested exception that
// the `runtimeConfig` from the network configuration is included
// here.
type RuntimeConf struct {
	ContainerID string
	NetNS       string
	IfName      string

[justxuewei]

We may use RuntimeConf.ContainerID instead of netns, so that we can support network without creating a new netns.

Sounds great to me. One of the important things is to define the fields of the devconfig, which should consider the details of the networking carefully.

For "vhost-user", it might look like this.

{
	"type": "vhost-user",
	"devconfig": {
		"path": "/path/to/vhost-user-socket",
		"server-mode": true
	}
}

where the path denotes the path of vhost-user socket, and the server-mode indicates whether the hypervisor is running in the server mode.

For "host-tap", it might look like this.

{
	"type": "host-tap",
	"devconfig": {
		"link": 1
	}
}

where the link indicates the link index of the tap device in the host netns.

[mzweilz]

Hi community,

I would like to give a feasible idea for this issue. As discussed in this thread and #4914, the tap devices in the host netns and vhost-user sockets should be attached to the hypervisor as well. Setting up dummy devices in the netns as the Kata containers 2.0 does is not a good idea. We use a file, instead of dummy devices, to exchange the network information because of flexibility and expandability.

The first question: Where are those network configs stored?

The name of netns identifies the network configs of a pod. Assumed that we have a netns, named cni-725a6170-996e-63df-2b80-4e5d75fba6e2, and the default basepath is /var/lib/cni/netconf, which it could be configurable. Eventually, Kata containers will load the network configs at /var/lib/cni/netconf/cni-725a6170-996e-63df-2b80-4e5d75fba6e2.

The second question: What is the format of the network configs?

There are two types at least, vhost-user-socket and tap in the host netns, to be supported. The content of the file is an array in the format of json as shown in the following.

[{
	"name": "example",
	// the possible values are "vhost-user" or "host-tap"
	"type": "vhost-user",
	// !!todo: to be determined
	"dev_conf": {
		"path": "/var/run/openvswitch/vhost_sockets/sock"
	},
	"network_info": {
		"interface": {
			"ip_addrs": [{
				"family": "v4",
				"address": "192.168.1.1",
				"mask": "16"
			}],
			"device": "device_name",
			"hardware_addr": "xx:xx:xx:xx:xx",
			"mtu": 1500,
			"raw_flags": "0x11"
		},
		"routers": [{
			"dest": "172.18.0.0/16",
			"src": "172.18.0.1",
			"gw": "172.18.31.1",
			"scope": 1
		}],
		"neighbors": [{
			"to_ip_addr": {
				"family": "v4",
				"address": "192.168.1.1",
				"mask": "16"
			},
			"state": 0,
			"flags": 0,
			"hardware_addr": "xx:xx:xx:xx:xx"
		}]
	}
}]

The last question: How do the Kata containers collaborate with the CNI plugins to set up the network?

The CNI plugins should set up net devices and save the network configs into the path we mentioned above. Before scanning the network devices in the netns, the Kata containers will load the json config and set up corresponding endpoints if the file exists.

The process of setting up the network looks like this:

Hi, it seems that when using the 'vhost-user' network type, the 'dev_conf' section in the configuration JSON file should also include 'queue_num' and 'queue_size' fields. Additionally, a 'mode' field is also expected if we want to configure the frontend as either a client or server. Could you please add the above support?

[justxuewei]

Hi, it seems that when using the 'vhost-user' network type, the 'dev_conf' section in the configuration JSON file should also include 'queue_num' and 'queue_size' fields. Additionally, a 'mode' field is also expected if we want to configure the frontend as either a client or server. Could you please add the above support?

@mzweilz Thanks for providing information! So, I would like to give two JSON templates for vhost-user and host-tap respectively. Please feel free to let me know if you have any question.

[{
	"device": {
		"type": "vhost-user",
		"path": "/var/run/openvswitch/vhost_sockets/sock",
		"queue_num": 1,
		"queue_size": 1,
		// the possible values are "server" and "client", the default is "server"
		"mode": "server"
	},
	"network_info": {
		"interface": {
			"ip_addresses": ["192.168.0.1/16", "192.168.0.3/16"],
			"mtu": 1500,
			"flags": 0,
			"ntype": "tuntap"
		},
		"routers": [{
			"dest": "172.18.0.0/16",
			"source": "172.18.0.1",
			"gateway": "172.18.31.1",
			"scope": 0,
			"flags": 0,
		}],
		"neighbors": [{
			"ip_address": "192.168.0.3/16",
			"state": 0,
			"flags": 0,
			"hardware_addr": "xx:xx:xx:xx:xx"
		}]
	}
},
{
	"device": {
		"type": "host-tap",
		"name": "dantap0"
	},
	"network_info": {
		// snipped
	}
}]