Problem with io_uring_prep_connect and Bluetooth sockets

[AidanSun05]

I'm rewriting an application that has both Internet and Bluetooth sockets to use io_uring. While I had success with using the liburing functions with Internet sockets, I'm facing some issues with Bluetooth ones.

The program's flow is: socket => io_uring_prep_connect => io_uring_prep_{send,recv} loop => close.

With a TCP or UDP socket, it works without errors; with a Bluetooth RFCOMM socket, io_uring_prep_connect fails and returns -77 (EBADF EBADFD) in cqe->res. However, it seems the connection actually succeeded, because subsequent *_send/*_recv calls have a positive cqe->res value.

Reproducible example:

#include <stdio.h>
#include <errno.h>

#include <liburing.h>
#include <unistd.h>
#include <sys/socket.h>
#include <bluetooth/bluetooth.h>
#include <bluetooth/rfcomm.h>

int connectSocket(struct io_uring* ring, int s) {
  bdaddr_t bdaddr;
  str2ba("3C:61:05:2A:ED:12", &bdaddr); // <=============== (You might need to change this)
  struct sockaddr_rc addr = { AF_BLUETOOTH, bdaddr, 1 };

#ifdef PLAIN_CONNECT
  return connect(s, (struct sockaddr*)&addr, sizeof(addr));
#else
  struct io_uring_sqe* sqe = io_uring_get_sqe(ring);
  io_uring_prep_connect(sqe, s, (struct sockaddr*)&addr, sizeof(addr));
  io_uring_submit(ring);

  struct io_uring_cqe* cqe;
  io_uring_wait_cqe(ring, &cqe);

  int res = cqe->res;
  io_uring_cqe_seen(ring, cqe);

  return res;
#endif
}

int sendData(struct io_uring* ring, int s, const char* data, int dataLen) {
  struct io_uring_sqe* sqe = io_uring_get_sqe(ring);
  io_uring_prep_send(sqe, s, data, dataLen, 0);
  io_uring_submit(ring);

  struct io_uring_cqe* cqe;
  io_uring_wait_cqe(ring, &cqe);

  int res = cqe->res;
  io_uring_cqe_seen(ring, cqe);

  return res;
}

int main() {
  struct io_uring ring;
  io_uring_queue_init(128, &ring, 0);

  int s = socket(AF_BLUETOOTH, SOCK_STREAM, BTPROTO_RFCOMM);
  printf("Socket file descriptor is: %d\n", s);

  int connectResult = connectSocket(&ring, s);
  printf("Connect result is: %d\n", connectResult);

  int sendResult = sendData(&ring, s, "Test", 4);
  printf("Send result is: %d\n", sendResult);

  close(s);
  io_uring_queue_exit(&ring);
  return 0;
}

$ gcc connect_test.c -luring -lbluetooth && ./a.out
Socket file descriptor is: 4
Connect result is: -77
Send result is: 4
$ gcc connect_test.c -DPLAIN_CONNECT -luring -lbluetooth && ./a.out
Socket file descriptor is: 4
Connect result is: 0
Send result is: 4

Does io_uring support Bluetooth sockets, or is there something extra I need to do in my code? While this currently "works", I feel like simply ignoring the error isn't a good solution. I would appreciate it if someone pointed me in the right direction.

---

liburing version: 2.2

Output of uname -a on two systems I tested on:

Fedora 36: Linux fedora 5.19.11-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Sep 23 15:07:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Arch Linux: Linux archlinux 5.19.13-arch1-1 #1 SMP PREEMPT_DYNAMIC Tue, 04 Oct 2022 14:36:58 +0000 x86_64 GNU/Linux

[axboe]

-77 is EBADFD, not EBADF. There's really no way for that to happen unless your liburing or something is really messed up, or can happen if the ring is setup disabled. My guess would be that you have a mismatch between the liburing headers and the library it links against.

[axboe]

Would be useful to include:

1. strace of running the program
2. separate from 1, run with io_uring tracing enabled

To trace what's happening on the io_uring side, do:

# cd /sys/kernel/debug/tracing
# echo 1 > events/io_uring/enable

run your app

# cat trace > /tmp/somefile

and then attach somefile in here.

[AidanSun05]

There's really no way for that to happen unless your liburing or something is really messed up, or can happen if the ring is setup disabled. My guess would be that you have a mismatch between the liburing headers and the library it links against.

I tested with liburing from both Conan and my system's package manager (both ran with the same result). I made sure to isolate them so I wasn't including the headers of one and linking to the other, so I believe it's not a library issue.

strace output:

execve("./a.out", ["./a.out"], 0x7ffe478bcd40 /* 54 vars */) = 0
brk(NULL)                               = 0x55b6b3010000
arch_prctl(0x3001 /* ARCH_??? */, 0x7fffb6256a70) = -1 EINVAL (Invalid argument)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe6fc422000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=93135, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 93135, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fe6fc40b000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/liburing.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=22392, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 24592, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe6fc404000
mmap(0x7fe6fc406000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fe6fc406000
mmap(0x7fe6fc408000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7fe6fc408000
mmap(0x7fe6fc409000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7fe6fc409000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libbluetooth.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=197896, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 196072, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe6fc3d4000
mmap(0x7fe6fc3dc000, 69632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7fe6fc3dc000
mmap(0x7fe6fc3ed000, 81920, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7fe6fc3ed000
mmap(0x7fe6fc401000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2d000) = 0x7fe6fc401000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P4\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1953472, ...}, AT_EMPTY_PATH) = 0
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 1994384, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe6fc1ed000
mmap(0x7fe6fc20f000, 1421312, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7fe6fc20f000
mmap(0x7fe6fc36a000, 356352, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17d000) = 0x7fe6fc36a000
mmap(0x7fe6fc3c1000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d4000) = 0x7fe6fc3c1000
mmap(0x7fe6fc3c7000, 52880, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe6fc3c7000
close(3)                                = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe6fc1ea000
arch_prctl(ARCH_SET_FS, 0x7fe6fc1ea740) = 0
set_tid_address(0x7fe6fc1eaa10)         = 2820
set_robust_list(0x7fe6fc1eaa20, 24)     = 0
rseq(0x7fe6fc1eb060, 0x20, 0, 0x53053053) = 0
mprotect(0x7fe6fc3c1000, 16384, PROT_READ) = 0
mprotect(0x7fe6fc401000, 4096, PROT_READ) = 0
mprotect(0x7fe6fc409000, 4096, PROT_READ) = 0
mprotect(0x55b6b2619000, 4096, PROT_READ) = 0
mprotect(0x7fe6fc455000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
munmap(0x7fe6fc40b000, 93135)           = 0
io_uring_setup(128, {flags=0, sq_thread_cpu=0, sq_thread_idle=0, sq_entries=128, cq_entries=256, features=IORING_FEAT_SINGLE_MMAP|IORING_FEAT_NODROP|IORING_FEAT_SUBMIT_STABLE|IORING_FEAT_RW_CUR_POS|IORING_FEAT_CUR_PERSONALITY|IORING_FEAT_FAST_POLL|IORING_FEAT_POLL_32BITS|IORING_FEAT_SQPOLL_NONFIXED|IORING_FEAT_EXT_ARG|IORING_FEAT_NATIVE_WORKERS|IORING_FEAT_RSRC_TAGS|IORING_FEAT_CQE_SKIP|IORING_FEAT_LINKED_FILE, sq_off={head=0, tail=64, ring_mask=256, ring_entries=264, flags=276, dropped=272, array=4416}, cq_off={head=128, tail=192, ring_mask=260, ring_entries=268, overflow=284, cqes=320, flags=280}}) = 3
mmap(NULL, 4928, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 3, 0) = 0x7fe6fc420000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 3, 0x10000000) = 0x7fe6fc41e000
socket(AF_BLUETOOTH, SOCK_STREAM, BTPROTO_RFCOMM) = 4
newfstatat(1, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x1), ...}, AT_EMPTY_PATH) = 0
getrandom("\xc5\x7f\x34\x3d\x1b\x98\x7b\xa6", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x55b6b3010000
brk(0x55b6b3031000)                     = 0x55b6b3031000
write(1, "Socket file descriptor is: 4\n", 29Socket file descriptor is: 4
) = 29
io_uring_enter(3, 1, 0, 0, NULL, 8)     = 1
io_uring_enter(3, 0, 1, IORING_ENTER_GETEVENTS, NULL, 8) = 0
write(1, "Connect result is: -77\n", 23Connect result is: -77
) = 23
io_uring_enter(3, 1, 0, 0, NULL, 8)     = 1
write(1, "Send result is: 4\n", 18Send result is: 4
)     = 18
close(4)                                = 0
munmap(0x7fe6fc41e000, 8192)            = 0
munmap(0x7fe6fc420000, 4928)            = 0
close(3)                                = 0
exit_group(0)                           = ?

io_uring trace:

# tracer: nop
#
# entries-in-buffer/entries-written: 10/10   #P:4
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
           a.out-2869    [003] .....  1787.445694: io_uring_create: ring 00000000a0f5499f, fd 3 sq size 128, cq size 256, flags 0x0
           a.out-2869    [003] .....  1787.446048: io_uring_submit_sqe: ring 00000000a0f5499f, req 0000000016e62989, user_data 0x0, opcode CONNECT, flags 0x0, non block 1, sq_thread 0
           a.out-2869    [003] .....  1787.446049: io_uring_file_get: ring 00000000a0f5499f, req 0000000016e62989, user_data 0x0, fd 4
           a.out-2869    [003] .....  1787.446067: io_uring_poll_arm: ring 00000000a0f5499f, req 0000000016e62989, user_data 0x0, opcode CONNECT, mask 0x4000010e, events 0x4000213e
           a.out-2869    [001] .....  1787.446105: io_uring_cqring_wait: ring 00000000a0f5499f, min_events 1
     kworker/2:5-1537    [002] d..1.  1788.484147: io_uring_task_add: ring 00000000a0f5499f, req 0000000016e62989, user_data 0x0, opcode CONNECT, mask 0
           a.out-2869    [001] ...1.  1788.484207: io_uring_complete: ring 00000000a0f5499f, req 0000000016e62989, user_data 0x0, result -77, cflags 0x0 extra1 0 extra2 0 
           a.out-2869    [001] .....  1788.484530: io_uring_submit_sqe: ring 00000000a0f5499f, req 0000000016e62989, user_data 0x0, opcode SEND, flags 0x0, non block 1, sq_thread 0
           a.out-2869    [001] .....  1788.484531: io_uring_file_get: ring 00000000a0f5499f, req 0000000016e62989, user_data 0x0, fd 4
           a.out-2869    [001] ...1.  1788.484538: io_uring_complete: ring 00000000a0f5499f, req 0000000016e62989, user_data 0x0, result 4, cflags 0x0 extra1 0 extra2 0

[axboe]

The -EBADFD is indeed coming in via the CQE, and it's coming from bluetooth:

if (sk->sk_state != BT_OPEN && sk->sk_state != BT_BOUND) {
		err = -EBADFD;
		goto done;
	}

which I'm not quite sure how that would happen. Or at least how it'd be different between using io_uring to do the connect or normal connect(2).

Maybe I can see if I can reproduce this. I don't really have any bluetooth, I wonder if it's possible to do in qemu.

[axboe]

OK I think I got it. Seems to work for me now:

axboe@m1max ~> gcc -Wall -O2 -o b b.c -luring -lbluetooth                      2.092s
axboe@m1max ~> ./b                                                             0.096s
Socket file descriptor is: 4
Connect result is: 0
Send result is: 4
axboe@m1max ~> gcc -Wall -O2 -DPLAIN_CONNECT -o b b.c -luring -lbluetooth      3.842s
axboe@m1max ~> ./b                                                             0.045s
Socket file descriptor is: 4
Connect result is: 0
Send result is: 4
axboe@m1max ~>                                                                 3.766s

I'll attach a patch. It's a kernel patch, so not sure how comfortable you are with getting those applied and rebuilding the kernel...

[axboe]

connect-einprogress.txt
connect-einprogress-v5.19.txt

One is against current -git, the other one is against 5.19-stable.

[AidanSun05]

Thank you for your quick responses! I'll apply the patch and report back the results.

[axboe]

Sounds good, thanks. And thanks for the reproducer, makes things so much easier...

If you want a Reported-by: attribution in the patch, I'd be happy to add that too. I just need your real name and email for that. If you prefer just to link to this issue, that's fine too.

[AidanSun05]

If you want a Reported-by: attribution in the patch, I'd be happy to add that too. I just need your real name and email for that.

Sounds great! My real name is Aidan Sun and my email is aidansun05@gmail.com.

[axboe]

Perfect, thanks! Here's the upstream commit, as queued:

https://git.kernel.dk/cgit/linux-block/commit/?h=for-6.1/io_uring-late&id=521aacebab32c79bcf985530a0130357e3d7d1c9

Slightly optimized from the ones I dumped here, but both should get the job done.

[AidanSun05]

I got the latest kernel from git and I applied the patch, and I can confirm it works.

aidan@aidan-VirtualBox:~$ gcc connect_test.c -luring -lbluetooth && ./a.out
Socket file descriptor is: 4
Connect result is: 0
Send result is: 4
aidan@aidan-VirtualBox:~$ gcc connect_test.c -DPLAIN_CONNECT -luring -lbluetooth && ./a.out
Socket file descriptor is: 4
Connect result is: 0
Send result is: 4
aidan@aidan-VirtualBox:~$ uname -a
Linux aidan-VirtualBox 6.0.0+ #1 SMP PREEMPT_DYNAMIC Wed Oct 5 14:09:36 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
aidan@aidan-VirtualBox:~$

[axboe]

Excellent! Thanks for the report and for testing the fix.

[axboe]

Closing this one up - the fix will trickle down to stable kernels as well, should go upstream soon.