io_uring: don't assume completions order

[ianic]

We are posting two submissions (zero copy send and receive) and then reading two completions. There is no guarantee that those completions will be in the order of submissions.
This test was expecting fist send completion then receive. Fix is allowing them to come the other way too.

Fixes: #20212

[andrewrk]

Thanks!

[andrewrk]

Still getting failures:

[nix-shell:~/dev/zig/build-release]$ uname -a
Linux bark 6.9.2 #1-NixOS SMP PREEMPT_DYNAMIC Sat May 25 14:30:56 UTC 2024 x86_64 GNU/Linux
[nix-shell:~/dev/zig/build-release]$ stage3/bin/zig build test-std
test-std
└─ run test std-x86_64-linux.4.19...6.5.7-none-x86_64-Debug 2724/2760 passed, 1 failed, 35 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
/home/andy/dev/zig/lib/std/testing.zig:93:17: 0x1e2b59c in expectEqualInner__anon_14403 (test)
                return error.TestExpectedEqual;
                ^
/home/andy/dev/zig/lib/std/testing.zig:134:17: 0x2cb5ba1 in expectEqualInner__anon_54300 (test)
                try expectEqual(@field(expected, field.name), @field(actual, field.name));
                ^
/home/andy/dev/zig/lib/std/os/linux/IoUring.zig:3544:5: 0x2ccf6e6 in test.accept/connect/send_zc/recv (test)
    try testing.expectEqual(linux.io_uring_cqe{
    ^
error: while executing test 'http.Server.Request.Head.decltest.parse', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/ce13529c13afb2a016647baae0dcfe2b/test --listen=- 
test-std
└─ run test std-x86-linux.4.19...6.5.7-gnu.2.28-pentium4-Debug-libc failure
Could not start dynamically linked executable: /home/andy/dev/zig/.zig-cache/o/c4af8459e29fb62b5e43e3b08fb8472d/test
NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box. For more information, see:
https://nix.dev/permalink/stub-ld
error: the following command exited with code 127 (expected exited with code 0):
/home/andy/dev/zig/.zig-cache/o/c4af8459e29fb62b5e43e3b08fb8472d/test --listen=- 
test-std
└─ run test std-x86-linux.4.19...6.5.7-none-pentium4-Debug 2722/2760 passed, 1 failed, 37 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
/home/andy/dev/zig/lib/std/testing.zig:93:17: 0xdd12e0 in expectEqualInner__anon_14398 (test)
                return error.TestExpectedEqual;
                ^
/home/andy/dev/zig/lib/std/testing.zig:134:17: 0x1c6e2f5 in expectEqualInner__anon_54150 (test)
                try expectEqual(@field(expected, field.name), @field(actual, field.name));
                ^
/home/andy/dev/zig/lib/std/os/linux/IoUring.zig:3544:5: 0x1c88283 in test.accept/connect/send_zc/recv (test)
    try testing.expectEqual(linux.io_uring_cqe{
    ^
error: while executing test 'http.Server.Request.Head.decltest.parse', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/6cc5eef2cb15949bccb0afcfbd919aa5/test --listen=- 
test-std
└─ run test std-x86_64-linux.4.19...6.5.7-musl-x86_64-Debug-libc-no-lld 2724/2760 passed, 1 failed, 35 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
/home/andy/dev/zig/lib/std/testing.zig:93:17: 0x1e5307b in expectEqualInner__anon_14486 (test)
                return error.TestExpectedEqual;
                ^
/home/andy/dev/zig/lib/std/testing.zig:134:17: 0x2e03d61 in expectEqualInner__anon_54101 (test)
                try expectEqual(@field(expected, field.name), @field(actual, field.name));
                ^
/home/andy/dev/zig/lib/std/os/linux/IoUring.zig:3544:5: 0x2e220f4 in test.accept/connect/send_zc/recv (test)
    try testing.expectEqual(linux.io_uring_cqe{
    ^
error: while executing test 'http.Server.Request.Head.decltest.parse', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/4241d3a63b3ed4ac49b815e30e33927b/test --listen=- 
test-std
└─ run test std-x86-linux.4.19...6.5.7-musl-pentium4-Debug-libc 2722/2760 passed, 1 failed, 37 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
/home/andy/dev/zig/lib/std/testing.zig:93:17: 0xdf3d69 in expectEqualInner__anon_14480 (test)
                return error.TestExpectedEqual;
                ^
/home/andy/dev/zig/lib/std/testing.zig:134:17: 0x1d751d5 in expectEqualInner__anon_53935 (test)
                try expectEqual(@field(expected, field.name), @field(actual, field.name));
                ^
/home/andy/dev/zig/lib/std/os/linux/IoUring.zig:3544:5: 0x1d9297e in test.accept/connect/send_zc/recv (test)
    try testing.expectEqual(linux.io_uring_cqe{
    ^
error: while executing test 'http.Server.Request.Head.decltest.parse', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/0f7622a48af12d0eb7270f11a7693488/test --listen=- 
test-std
└─ run test std-x86_64-linux.4.19...6.5.7-musl-x86_64-Debug-libc 2724/2760 passed, 1 failed, 35 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
/home/andy/dev/zig/lib/std/testing.zig:93:17: 0x1e5307b in expectEqualInner__anon_14486 (test)
                return error.TestExpectedEqual;
                ^
/home/andy/dev/zig/lib/std/testing.zig:134:17: 0x2e03d61 in expectEqualInner__anon_54101 (test)
                try expectEqual(@field(expected, field.name), @field(actual, field.name));
                ^
/home/andy/dev/zig/lib/std/os/linux/IoUring.zig:3544:5: 0x2e220f4 in test.accept/connect/send_zc/recv (test)
    try testing.expectEqual(linux.io_uring_cqe{
    ^
error: while executing test 'http.Server.Request.Head.decltest.parse', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/4241d3a63b3ed4ac49b815e30e33927b/test --listen=- 
test-std
└─ run test std-x86_64-linux.4.19...6.5.7-none-x86_64_v2-Debug-selfhosted-no-lld-pic 2714/2760 passed, 1 failed, 45 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
error: while executing test 'http.Server.Request.Head.decltest.parse', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/12432743b4eb015f05b5af48585affc4/test --listen=- 
test-std
└─ run test std-x86_64-linux.6.9.2...6.9.2-gnu.2.39-znver4-Debug-single 2693/2758 passed, 1 failed, 64 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
/home/andy/dev/zig/lib/std/testing.zig:93:17: 0x1b3ce4c in expectEqualInner__anon_14323 (test)
                return error.TestExpectedEqual;
                ^
/home/andy/dev/zig/lib/std/testing.zig:134:17: 0x28acf31 in expectEqualInner__anon_53919 (test)
                try expectEqual(@field(expected, field.name), @field(actual, field.name));
                ^
/home/andy/dev/zig/lib/std/os/linux/IoUring.zig:3544:5: 0x28c6c5e in test.accept/connect/send_zc/recv (test)
    try testing.expectEqual(linux.io_uring_cqe{
    ^
error: while executing test 'compress.zstandard.decode.fse.decltest.buildFseTable', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/bebe89c7e94569e065ba09f7191db2ae/test --listen=- 
test-std
└─ run test std-x86_64-linux.4.19...6.5.7-none-x86_64-Debug-selfhosted-no-lld 2714/2760 passed, 1 failed, 45 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
error: while executing test 'http.Server.Request.Head.decltest.parse', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/10cdcd79730d16e10833e882da477a9a/test --listen=- 
test-std
└─ run test std-x86_64-linux.6.9.2...6.9.2-gnu.2.39-znver4-Debug 2724/2760 passed, 1 failed, 35 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
/home/andy/dev/zig/lib/std/testing.zig:93:17: 0x1e200ac in expectEqualInner__anon_14403 (test)
                return error.TestExpectedEqual;
                ^
/home/andy/dev/zig/lib/std/testing.zig:134:17: 0x2bba121 in expectEqualInner__anon_54283 (test)
                try expectEqual(@field(expected, field.name), @field(actual, field.name));
                ^
/home/andy/dev/zig/lib/std/os/linux/IoUring.zig:3544:5: 0x2bd3e4e in test.accept/connect/send_zc/recv (test)
    try testing.expectEqual(linux.io_uring_cqe{
    ^
error: while executing test 'http.Server.Request.Head.decltest.parse', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/c757731b02ece33b3df5d1adf2b51805/test --listen=- 
test-std
└─ run test std-x86_64-linux.4.19...6.5.7-none-x86_64_v3-Debug-selfhosted-no-lld 2714/2760 passed, 1 failed, 45 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
error: while executing test 'http.Server.Request.Head.decltest.parse', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/4908460056ba4e6c888026c0ac4bf168/test --listen=- 
test-std
└─ run test std-x86_64-linux.6.9.2...6.9.2-gnu.2.39-znver4-ReleaseSmall-single 2692/2758 passed, 1 failed, 65 skipped
error: 'os.linux.IoUring.test.accept/connect/send_zc/recv' failed: expected 4294967295, found 4008636142
error: while executing test 'compress.zstandard.decode.fse.decltest.buildFseTable', the following test command failed:
/home/andy/dev/zig/.zig-cache/o/c5cbf68fca6477ac87054d4cc57214c5/test --listen=- 

[mikdusan]

I am seeing this order; and in consideration of IOSQE_IO_LINK flag, iiuc, recv will for sure come after send1. I'm just not clear about send2 guarantee. It looks like if you change the patch block to assume send1 is first, and then sort send2/recv .

- cqe send1: 0xeeeeeeee <-- linux.IORING_CQE_F_MORE
- cqe send2: 0xeeeeeeee <-- linux.IORING_CQE_F_NOTIF
- cqe recv: 0xffffffff

[ianic]

Uf, sorry for wasting your time.

In the first try I didn't make an effort to reproduce bug on the required kernel and made wrong conclusion about what completions have different order on newer kernel. Here is new PR which is tested on 6.9.2 kernel.