Calling `io_uring_setup` from multiple threads

[tchaloupka]

Hi, I've probably found another problem triggered when my tests run in parallel.
Should it be ok to setup unique io_uring independently from multiple threads?

I've created simple test to reproduce it (at least on my Ryzen 7 3700X with Fedora 31 kernel 5.3.12).

Basically it fails during io_uring_setup call with ENOMEM.
I've tried to follow this guide to figure something out, but I'm not into kernel dev so this all seems too much woodoo for me ;-)

Anyway, here's the trace output if it helps somehow:

  1)               |  __x64_sys_io_uring_setup() {
  1)               |    io_uring_setup() {
  1)               |      capable() {
  1)               |        ns_capable_common() {
  1)  <...>-633600  =>  <...>-633604 
  1)               |          security_capable() {
  1)  <...>-633604  =>  <...>-633600 
  1)   0.230 us    |            cap_capable();
  1)   0.762 us    |          }
  1)   1.202 us    |        }
  1)   1.623 us    |      }
  1)   0.270 us    |      free_uid();
  1)   2.735 us    |    }
  1)   3.467 us    |  }

And here is a test to reproduce it.

#include <stdio.h>
#include <pthread.h>
#include "liburing.h"

struct thread_info_t {
	pthread_t tid;
	int num;
};

static void *doTest(void *arg) {
	struct io_uring ring;
	struct io_uring_cqe *cqe;
	struct io_uring_sqe *sqe;
	struct thread_info_t *ti;
	int ret;

	ti = (struct thread_info_t *)arg;
	printf("%d: start\n", ti->num);

	ret = io_uring_queue_init(128, &ring, 0);
	if (ret) {
		printf("%d: ring setup failed: %d\n", ti->num, ret);
		return arg;
	}

	sqe = io_uring_get_sqe(&ring);
	if (!sqe) {
		printf("%d: get sqe failed\n", ti->num);
		return arg;
	}

	io_uring_prep_nop(sqe);

	ret = io_uring_submit(&ring);
	if (ret <= 0) {
		printf("%d: sqe submit failed: %d\n", ti->num, ret);
		return arg;
	}

	ret = io_uring_wait_cqe(&ring, &cqe);
	if (ret < 0) {
		printf("%d: wait completion %d\n", ti->num, ret);
		return arg;
	}

	io_uring_cqe_seen(&ring, cqe);
	printf("%d: done\n", ti->num);
	return NULL;
}

int main(int argc, char *argv[])
{
	struct thread_info_t threads[10];
	int ret;
	void *res;

	for (int i=0; i<10; i++) {
		threads[i].num = i;
		ret = pthread_create(&threads[i].tid, NULL, doTest, &threads[i]);
		if (ret) {
			fprintf(stderr, "Thread create failed\n");
			return 1;
		}
	}

	for (int i=0; i<10; i++) {
		ret = pthread_join(threads[i].tid, &res);
		if (ret) {
			fprintf(stderr, "Thread join failed\n");
			return 1;
		}
		if (res) {
			fprintf(stderr, "Test failed\n");
			return 1;
		}
	}

	return 0;
}

One of my outputs is:

0: start
1: start
2: start
3: start
4: start
4: ring setup failed: -12
5: start
5: ring setup failed: -12
6: start
0: done
6: ring setup failed: -12
3: done
7: start
7: ring setup failed: -12
1: done
8: start
8: ring setup failed: -12
9: start
9: ring setup failed: -12
2: done
Test failed

[axboe]

It's not a thread problem, I'm guessing you are running as non-root and you run into the rlimit memlock limit (ulimit -l).

[tchaloupka]

Ah, thanks, I've totally overlooked that.
Test runs fine when run as root.