When does io_uring create tasks or otherwise start iou-wrk threads?

[armanbilge]

I am seeking more detailed information about when submitted I/O operations create tasks that are executed on the io_uring worker pool. This blog post has been a good resource, but it doesn't answer all of my questions.

My motivating "case study":

In armanbilge/fs2-io_uring#56 I created a TCP echo server with io_uring_prep_recv and io_uring_prep_send. I was surprised to discover that when running it spawns a number of iou-wrk threads, which I assumed should not be necessary for non-blocking network I/O.

So in armanbilge/fs2-io_uring#61 I added a io_uring_prep_poll_add that then IOSQE_IO_LINKs to the recv/send. After making this change, it appears that my echo server no longer spawns iou-wrk threads.

I'd really like to understand why that is and how I can be more conscious in the future about how to submit I/O that does not require the worker pool, if possible. For example, I am wondering if I should make a similar change for accept.

Thanks in advance!

[isilence]

That shouldn't happen, not often, and it should do fine if you just submit a send/recv without polling first.

What is your kernel version? Is it nonblocking (SOCK_NONBLOCK, MSG_DONTWAIT)? Do you use MSG_WAITALL?

[armanbilge]

Here's another version, which doesn't create an iou-wrk thread on 6.1, but it does on 5.15.

Details

#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <stdlib.h>
#include <string.h>
#include <netinet/udp.h>
#include <arpa/inet.h>
#include <netdb.h>

#include <liburing.h>

int main(int argc, char *argv[])
{
  struct io_uring ring;
  io_uring_queue_init(8, &ring, 0);
  int fd = socket(AF_INET, SOCK_STREAM, 0);
  struct addrinfo * ai;
  getaddrinfo("httpbin.org", "80", NULL, &ai);
  connect(fd, ai->ai_addr, ai->ai_addrlen);
  char* req = "GET /get HTTP/1.1\nHost: httpbin.org\n\n";
  
  while (true) {
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    struct io_uring_cqe *cqe;
    io_uring_prep_send(sqe, fd, req, strlen(req), MSG_NOSIGNAL);
    
    int ret = io_uring_submit(&ring);
    if (ret != 1) {
      perror("io_uring_submit");
      return 1;
    }
    
    ret = io_uring_wait_cqe(&ring, &cqe);
    if (ret < 0) {
      perror("io_uring_wait_cqe");
      return 1;
    }
    
    io_uring_cqe_seen(&ring, cqe);
  }
}

[isilence]

I grabbed a fresh stable 5.15 but couldn't reproduce, not sure what's the difference.
Where did you get the kernel from? Did you compile it yourself or was it provided by a Linux distribution?

[armanbilge]

Huh, thanks for trying. It's a 5.15.0-71-generic kernel that I got with an Ubuntu 22.04 LTS server. I just checked again now and I can still reproduce. It took maybe 30 seconds before the iou-wrk thread appeared.

[isilence]

I was running it for an hour, nothing, but it was qemu. I asked about the kernel because distributions apply random kernel patches, so it might be different.

[axboe]

@armanbilge I tried this with 5.15-stable (5.15.117) and with 5.15.0-75-generic and can't seem to reproduce it either. Assuming you updated to -75 as well, does it happen for you there?

[axboe]

What kernel is being used?

[isilence]

See above, 5.15 and @armanbilge tried out 6.2, which should be before retries and APOLL_MAX_RETRY came in.