Support io_uring's IORING_SETUP_SQPOLL flag

[tatetian]

Motivation

To avoid exiting enclaves when submitting I/O requests, we want to push requests into io_uring's submission queue (by memory copying, thus no enclave switching) and let the Linux kernel poll requests from the submission queue (also by memory copying, thus no enclave switching). The latter is enabled with io_uring's IORING_SETUP_SQPOLL flag.

Problem

The problem is that the current implementation does not use IORING_SETUP_SQPOLL flag. This is because at the time when the code was written, Linux kernel's support for the submission queue polling was incomplete and buggy. So as a workaround, the implementation starts a separate OS thread that keeps invoking io_uring_enter system call. The workaround fulfils our demands of switchless I/O but do so at the expense of wasting CPU cycles.

Solution

As Linux's io_uring implementation gets mature, IORING_SETUP_SQPOLL flag now becomes fully supported since Linux kernel 5.11. And our development machine is installed with Linux kernel 5.11. So it is time for us to ditch the workaround and leverage IORING_SETUP_SQPOLL flag. Specifically, we need to

• Upgrade the occlum/io-uring crate
  • The crate is a fork of tokio-rs/io-uring, with modifications to support SGX. The original tokio-rs/io-uring crate has added support for IORING_SETUP_SQPOLL.
  • Need to rebase our fork to the latest master branch of tokio-rs/io-uring
  • Make sure all unit tests pass, including the SGX sample code
• Use the new occlum/io-uring in LibOS and other crates
  • Add the IORING_SETUP_SQPOLL flag when creating an io_uring instance
  • Set the sq_thread_idle parameter to a proper value, e.g., 500 milliseconds
  • Remove the workaround
• Document the minimal requirement for Linux kernel is 5.11

[tatetian]

It turns out that the current version of occlum/io-uring already includes the support for IORING_SETUP_SQPOLL. I tested it. And it worked. So I am going to close this issue.