Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime-rs: improving io performance using dragonball's vsock fd passthrough #7483

Merged
merged 16 commits into from
Feb 1, 2024
Merged

runtime-rs: improving io performance using dragonball's vsock fd passthrough #7483

merged 16 commits into from
Feb 1, 2024

Commits on Jan 31, 2024

  1. runtime-rs: make Container::new async 

    Fixes: kata-containers#6714

Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    973b5ad View commit details
    Browse the repository at this point in the history

  2. config: add two options to control vsock passthrough io feature 

    Two toml options, `use_passfd_io` and `passfd_listener_port` are introduced
to enable and configure dragonball's vsock fd passthrough io feature.

This commit is a preparation for vsock fd passthrough io feature.

Fixes: kata-containers#6714

Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    eb6bb6f View commit details
    Browse the repository at this point in the history

  3. agent,runtime-rs: refactor process io using vsock fd passthrough feature 

    Currently in the kata container, every io read/write operation requires
an RPC request from the runtime to the agent. This process involves
data copying into/from an RPC request/response, which are high overhead.

To solve this issue, this commit utilize the vsock fd passthrough, a
newly introduced feature in the Dragonball hypervisor. This feature
allows other host programs to pass a file descriptor to the Dragonball
process, directly as the backend of an ordinary hybrid vsock connection.

The runtime-rs now utilizes this feature for container process io. It
open the stdin/stdout/stderr fifo from containerd, and pass them to
Dragonball, then don't bother with process io any more, eliminating
the need for an RPC for each io read/write operation.

In passfd io mode, the agent uses the vsock connections as the child
process's stdin/stdout/stderr, eliminating the need for a pipe
to bump data (in non-tty mode).

Fixes: kata-containers#6714

Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    442df71 View commit details
    Browse the repository at this point in the history

  4. dragonball: vsock: properly handle EPOLLHUP/EPOLLERR events 

    When one end of the connection close, the epoll event will be triggered
forever. We should close the connection and kill the connection.

Fixes: kata-containers#6714

Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    b8632b4 View commit details
    Browse the repository at this point in the history

  5. agent: clean up term master fd when container exits 

    When container exits, the agent should clean up the term master fd,
otherwise the fd will be leaked.

Fixes: kata-containers#6714

Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    f1b33fd View commit details
    Browse the repository at this point in the history

  6. runtime-rs: open stdin fifo with RDWR|NONBLOCK when pass vsock streams 

    In linux, when a FIFO is opened and there are no writers, the reader
will continuously receive the HUP event. This can be problematic
when creating containers in detached mode, as the stdin FIFO writer
is closed after the container is created, resulting in this situation.

In passfd io mode, open stdin fifo with O_RDWR|O_NONBLOCK to avoid the
HUP event.

Fixes: kata-containers#6714
Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    657b17a View commit details
    Browse the repository at this point in the history

  7. agent,runtime-rs: fix container io detach and attach 

    Partially fix some issues related to container io detach and attach.

Fixes: kata-containers#6714
Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    5536743 View commit details
    Browse the repository at this point in the history

  8. dbs: hybrid stream support keep the connection when local closed 

    Support the hybrid fd passthrough mode with passing pipe fd,
which can specify this connection kept even when the pipe
peer closed, and this connection can be reget wich re-opening
the pipe.

Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
    @frezcirno
    Fupan Li authored and frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    4a762fc View commit details
    Browse the repository at this point in the history

  9. container: keep the io connection when pass fd to hybrid vsock 

    We want the io connection keep connected when the containerd closed
the io pipe, thus it can be attached on the io stream.

Signed-off-by: Fupan Li <fupan.lfp@antgroup.com>
    @frezcirno
    Fupan Li authored and frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    cfb262d View commit details
    Browse the repository at this point in the history

  10. agent: set stdout/err vsock stream as blocking before passing to child 

    In passfd io mode, when not using a terminal, the stdout/stderr vsock
streams are directly used as the stdout/stderr of the child process.
These streams are non-blocking by default.

The stdout/stderr of the process should be blocking, otherwise
the process may encounter EAGAIN error when writing to stdout/stderr.

Fixes: kata-containers#6714
Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    7874ef5 View commit details
    Browse the repository at this point in the history

  11. agent: use biased select to avoid data loss 

    This patch uses a biased select to avoid stdin data loss in case of
CloseStdinRequest.

Fixes: kata-containers#6714
Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    3eb4bed View commit details
    Browse the repository at this point in the history

  12. runtime-rs: open stdout and stderr fifos NONBLOCK 

    This patch adds O_NONBLOCK flag when open stdout and stderr FIFOs
to avoid blocking.

Fixes: kata-containers#6714
Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    89be42a View commit details
    Browse the repository at this point in the history

  13. agent,runtime-rs,runk: fix fmt and clippy warnings 

    Fix rustfmt and clippy warnings detected by CI.

Fixes: kata-containers#6714
Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    f671061 View commit details
    Browse the repository at this point in the history

  14. agent: Use pipes as stdout/stderr of container process 

    Linux forbids opening an existing socket through /proc/<pid>/fd/<fd>,
making some images relying on the special file /dev/stdout(stderr),
/proc/self/fd/1(2) fail to boot in passfd io mode, where the
stdout/stderr of a container process is a vsock socket.

For back compatibility, a pipe is introduced between the process
and the socket, and its read end is set as stdout/stderr of the
container process instead of the socket. The agent will do the
forwarding between the pipe and the socket.

Fixes: kata-containers#6714
Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    1206de2 View commit details
    Browse the repository at this point in the history

  15. agent,runtime-rs: Add license header to passfd_io.rs 

    Fixes: kata-containers#6714
Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    6e4d4c3 View commit details
    Browse the repository at this point in the history

  16. agent: Fix a race condition in passfd_io.rs 

    There is a race condition in agent HVSOCK_STREAMS hashmap, where a
stream may be taken before it is inserted into the hashmap. This patch
add simple retry logic to the stream consumer to alleviate this issue.

Fixes: kata-containers#6714
Signed-off-by: Zixuan Tan <tanzixuan.me@gmail.com>
    @frezcirno
    frezcirno committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    222de4f View commit details
    Browse the repository at this point in the history