SQPOLL: io_uring_submit return value greater than what's newly submitted

[wuxb45]

A strange behavior observed in recent io_uring usage.
When not using SQPOLL, io_uring_submit returns correct numbers of sqes recently filled.
WIth SQPOLL, the first submit seems to be correct. But in the future, when wait_cqe(), seen(), and submit() are mixed, submit() may return a greater number, such as 2, or 3 when submitting one sqe. But when waiting cqes, the numbers of received cqes (number of returned wait()s) matches what's expected, which is smaller than then sum of the submit()'s return values.
It seems the test/read-write.c also checks for number of submissions. But I have no idea what's wrong in my code.
The code used to work correctly but started to have this issue now. It's been a few month since that same code was last run. Not sure if it's a bug in my code or in uring.

The pseudo code here: If necessary I can post the (length) full code that reproduces the error. (Linux 5.5.6, liburing #e42430c)

for (i=0 to N) {
  get_sqe()
  prep...()
}
n = submit(); // n == N

for (i=0 to M) {
  wait()
  seen()
  get_sqe()
  submit() // this submit sometimes returns 2 or 3
}

for (i=0 to N) {
  wait()
  seen()
}
// will block if wait() one more.

[axboe]

Posting the full reproducer would be great, as it saves me time!

[wuxb45]

I doubled the ring size (to 2N) and it works without the problem.

[ashmrtn]

I have also observed this problem. I'm using the code at tag liburing-0.7, Ubuntu 18.04 VM and 5.6.0-rc4 kernel. I haven't created a minimal reproducer since I just now figured out this was the issue and was looking for other reports of it

[axboe]

Please do create one, it's the easiest way to discuss the issue as well. Thanks!

[ashmrtn]

I've created an minimal example with some error reporting here. I get that ideally you wouldn't submit sqes one at a time, but this is just to show the issue.

Output that I get when running this is something like the following:

ashmrtnz@vm:~$ ./io_uringExample /mnt/xfs/io_uringExample.txt
Submitted more sqes than expected: 2 (expected: 1)
Submitted more sqes than expected: 3 (expected: 1)
Submitted more sqes than expected: 4 (expected: 1)
Submitted more sqes than expected: 5 (expected: 1)
Submitted more sqes than expected: 6 (expected: 1)
Submitted more sqes than expected: 7 (expected: 1)
Submitted more sqes than expected: 8 (expected: 1)
Submitted more sqes than expected: 7 (expected: 1)
Submitted more sqes than expected: 8 (expected: 1)
After 10 queued, reported 51 as queued

If it makes any difference, this is running on top of an xfs file system.

[isilence]

SQPOLL is basically a parallel thread that taking out all SQEs
that it can reach. I don't think that trying to pass a submit number
is worth of complicating synchronisation.

[isilence]

Moreover, it can consume SQEs even if you don't do any syscalls (by polling).

[isilence]

Hmm, it looks like I've got the example wrong, my apologies. I'll try to run it.

[ashmrtn]

yes, I understand that it will consume SQEs without a syscall as long as the kernel thread hasn't been put to sleep (in which case a syscall to wake it will be required, though I believe liburing handles this?).

Whether or not io_uring_submit() should return meaningful values when using polling, the API documentation should probably be updated to reflect the fact. Right now, people who don't know the library well may assume that they need to wait for more requests than they submitted (based on the result of io_uring_submit()) which would cause problems to say the least

[isilence]

I've got the same results for the latest kernel. Nice reproducer!

syscall io_uring_enter() doesn't do much but returns @to_submit, which is
calculated as how many SQEs are currently in the SQ. Because asynchronous
SQPOLL thread does not consume SQEs immediately, the following
sys_io_uring_enter() (i.e. io_uring_submit()) may see previously filled entries
in the SQ.

In other words, if submission succeded, for SQPOLL the return value shows
how many SQEs were in the SQ, but not how many reqs were actually consumed.

[ashmrtn]

Makes sense, thanks for the explanation. I guess in this case it makes more sense for applications to use the result of io_uring_submit() just to make sure that it didn't fail completely and do their own tracking of how many requests have been created and submitted so they can check for completions. This is because, and correct me if I'm wrong, there's no way for and application to ask liburing for the number of in-flight operations that will eventually appear in the completion queue (and thus how many operations to wait on if they want to wait for all submitted I/O to complete)

[axboe]

Indeed, and it's really not possible to make this fully accurate. For SQPOLL, you'll have to retain this count yourself so you know how much is pending.

There is no way to ask for the number in-flight,