Kata Containers streaming IO #6714

bergwolf · 2023-04-25T03:06:37Z

Right now, we forward container io stream (stdin/stdout/stderr) wrapped with ttrpc request, which adds unnecessary overhead and is error-prone. We have been fighting with missing stdin/stdout bytes from time to time since the beginning of the project.

To solve it once and for all, let's add a streaming IO interface between runtime and agent, using vsock connections as the underlying data transfer channel. It can remove the runtime shim and the agent from container IO data path.

For example, for container stdio, we can go from

containerd pipefd <-> kata runtime shim <-> vmm <-> kata agent <-> container stdio pipefd

to

containerd pipefd <-> vmm <-> container stdio pipefd

Key change:

a new fd based vsock backend implementation, which forwards whatever data between the guest vsock device and the host fd
a new streaming IO agent API that helps to setup the fd connection between guest and host
runtime-rs/kata-agent change to help setup the fd mapping between guest and host
support container stdio with the streaming IO API
(possibly) portforward with the streaming IO API

The text was updated successfully, but these errors were encountered:

ClSlaid · 2023-05-31T08:17:29Z

Anyone taking on this?

bergwolf · 2023-06-06T09:16:31Z

@ClSlaid Yes. It has been assigned to a student in CCF summer of code.

Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Two toml options, `use_passfd_io` and `passfd_listener_port` are introduced to enable and configure dragonball's vsock fd passthrough io feature. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Currently in the kata container, every io read/write operation requires an RPC request from the runtime to the agent. This process involves unnecessary data copying into/from an RPC request/response, which introduces high overhead. To solve this issue, this commit utilize the vsock fd passthrough, a newly introduced feature in the Dragonball hypervisor. This feature allows other host programs to pass a file descriptor to the Dragonball process directly as the backend of an ordinary hybrid vsock connection. The runtime-rs now utilizes this feature for container process io. It passes the stdin/stdout/stderr fifo from containerd to Dragonball, eliminating the need for an RPC for each io read/write operation. The agent uses the vsock stream as the child process's stdin/stdout/stderr in passfd mode, eliminating the need for a pipe to bump data (in non-tty mode). Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Currently in the kata container, every io read/write operation requires an RPC request from the runtime to the agent. This process involves unnecessary data copying into/from an RPC request/response, which introduces high overhead. To solve this issue, this commit utilize the vsock fd passthrough, a newly introduced feature in the Dragonball hypervisor. This feature allows other host programs to pass a file descriptor to the Dragonball process directly as the backend of an ordinary hybrid vsock connection. The runtime-rs now utilizes this feature for container process io. It passes the stdin/stdout/stderr fifo from containerd to Dragonball, eliminating the need for an RPC for each io read/write operation. The agent uses the vsock stream as the child process's stdin/stdout/stderr in passfd mode, eliminating the need for a pipe to bump data (in non-tty mode). Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Two toml options, `use_passfd_io` and `passfd_listener_port` are introduced to enable and configure dragonball's vsock fd passthrough io feature. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Currently in the kata container, every io read/write operation requires an RPC request from the runtime to the agent. This process involves unnecessary data copying into/from an RPC request/response, which introduces high overhead. To solve this issue, this commit utilize the vsock fd passthrough, a newly introduced feature in the Dragonball hypervisor. This feature allows other host programs to pass a file descriptor to the Dragonball process directly as the backend of an ordinary hybrid vsock connection. The runtime-rs now utilizes this feature for container process io. It passes the stdin/stdout/stderr fifo from containerd to Dragonball, eliminating the need for an RPC for each io read/write operation. The agent uses the vsock stream as the child process's stdin/stdout/stderr in passfd mode, eliminating the need for a pipe to bump data (in non-tty mode). Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Two toml options, `use_passfd_io` and `passfd_listener_port` are introduced to enable and configure dragonball's vsock fd passthrough io feature. This commit is a preparation for vsock fd passthrough io feature. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Currently in the kata container, every io read/write operation requires an RPC request from the runtime to the agent. This process involves data copying into/from an RPC request/response, which are high overhead. To solve this issue, this commit utilize the vsock fd passthrough, a newly introduced feature in the Dragonball hypervisor. This feature allows other host programs to pass a file descriptor to the Dragonball process, directly as the backend of an ordinary hybrid vsock connection. The runtime-rs now utilizes this feature for container process io. It open the stdin/stdout/stderr fifo from containerd, and pass them to Dragonball, then don't bother with process io any more, eliminating the need for an RPC for each io read/write operation. In passfd io mode, the agent uses the vsock connections as the child process's stdin/stdout/stderr, eliminating the need for a pipe to bump data (in non-tty mode). Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

When one end of the connection close, the epoll event will be triggered forever. We should close the connection and kill the connection. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

In linux, when a FIFO is opened and there are no writers, the reader will continuously receive the HUP event. This can be problematic when creating containers in detached mode, as the stdin FIFO writer is closed after the container is created, resulting in this situation. In passfd io mode, open stdin fifo with O_RDWR|O_NONBLOCK to avoid the HUP event. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Partially fix some issues related to container io detach and attach. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Linux forbids opening an existing socket through /proc/<pid>/fd/<fd>, making some images relying on the special file /dev/stdout(stderr), /proc/self/fd/1(2) fail to boot in passfd io mode, where the stdout/stderr of a container process is a vsock socket. For back compatibility, a pipe is introduced between the process and the socket, and its read end is set as stdout/stderr of the container process instead of the socket. The agent will do the forwarding between the pipe and the socket. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

There is a race condition in agent HVSOCK_STREAMS hashmap, where a stream may be taken before it is inserted into the hashmap. This patch add simple retry logic to the stream consumer to alleviate this issue. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Two toml options, `use_passfd_io` and `passfd_listener_port` are introduced to enable and configure dragonball's vsock fd passthrough io feature. This commit is a preparation for vsock fd passthrough io feature. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Currently in the kata container, every io read/write operation requires an RPC request from the runtime to the agent. This process involves data copying into/from an RPC request/response, which are high overhead. To solve this issue, this commit utilize the vsock fd passthrough, a newly introduced feature in the Dragonball hypervisor. This feature allows other host programs to pass a file descriptor to the Dragonball process, directly as the backend of an ordinary hybrid vsock connection. The runtime-rs now utilizes this feature for container process io. It open the stdin/stdout/stderr fifo from containerd, and pass them to Dragonball, then don't bother with process io any more, eliminating the need for an RPC for each io read/write operation. In passfd io mode, the agent uses the vsock connections as the child process's stdin/stdout/stderr, eliminating the need for a pipe to bump data (in non-tty mode). Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

When one end of the connection close, the epoll event will be triggered forever. We should close the connection and kill the connection. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

In linux, when a FIFO is opened and there are no writers, the reader will continuously receive the HUP event. This can be problematic when creating containers in detached mode, as the stdin FIFO writer is closed after the container is created, resulting in this situation. In passfd io mode, open stdin fifo with O_RDWR|O_NONBLOCK to avoid the HUP event. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Partially fix some issues related to container io detach and attach. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

In passfd io mode, when not using a terminal, the stdout/stderr vsock streams are directly used as the stdout/stderr of the child process. These streams are non-blocking by default. The stdout/stderr of the process should be blocking, otherwise the process may encounter EAGAIN error when writing to stdout/stderr. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

This patch uses a biased select to avoid stdin data loss in case of CloseStdinRequest. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

This patch adds O_NONBLOCK flag when open stdout and stderr FIFOs to avoid blocking. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>

Fix rustfmt and clippy warnings detected by CI. Fixes: kata-containers#6714 Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>