[Linux] evaluate io_uring() as alternative async I/o mechanism

[hassila]

Stumbled across another (new) async I/o interface that seems worthwhile considering for best possible performance on Linux. See some background here:

https://lwn.net/Articles/810414/
https://kernel.dk/io_uring.pdf

And one simple test vs epoll with performance numbers:
https://github.com/frevib/io_uring-echo-server/blob/io-uring-op-provide-buffers/benchmarks/benchmarks.md

Is this something of interest?

[hassila]

Here are some results from netty who adopted it:
netty/netty#10622 (comment)

[PeterAdams-A]

Thanks for the heads up @hassila. io_uring has been on our radar for a while. Unfortunately it's yet to reach the top of our list as we are busy with many other things. We agree that it is likely to lead to improved performance on linux.

If you feel so inclined we'd welcome any code contributions in this direction.

[hassila]

Thanks @PeterAdams-A - I could have a look, it seems Ubuntu 20.04 has a fresh enough kernel that supports it (even if performance of io_uring seems to have improved quite a bit further with more recent ones than 5.4 that Ubuntu uses, but it should be usable for bringup).

Just a few basic initial questions (I'm sure I'd need some handholding on best integration with existing code-base):

• Looking at current #if usage, it seems that it is done on a platform rather than a feature basis - I'm used to (from previous life) to use e.g. ifdefs that checks for e.g. kqueue/epoll rather than OS. In this case, I don't think a PR should replace the current epoll implementation for multiple reasons (primarily support for older kernels on e.g. Amazon Linux 2 etc that lacks io_uring, but also eases of comparing performance between epoll/io_uring). What would be a best practice for supporting two implementations for a single platform and to be able to select between them?
• The inventor of io_uring also provides a convenience library (https://github.com/axboe/liburing) that removes a significant chunk of boilerplate - is this something that would be acceptable as a dependency / part of implementation, or would a "clean" use of the lower-level primitives (but with possibly harder-to-read code) be preferred for a successful PR?
• Would any of the existing tests be viewed as usable for validating performance, or how would you propose validation is done in an acceptable manner?

I'm sure there'll be additional questions, but just some stuff that came to mind when looking at it.

[Lukasa]

What would be a best practice for supporting two implementations for a single platform and to be able to select between them?

Selecting between them should just be configuration. They can hook in underneath SelectableEventLoop, I suspect. Swift does not have a preprocessor in the traditional sense so it cannot use #if-based feature detection.

The inventor of io_uring also provides a convenience library (https://github.com/axboe/liburing) that removes a significant chunk of boilerplate

There's no problem with relying on this library iff its absence can be tolerated on systems that do not have iouring.

Would any of the existing tests be viewed as usable for validating performance, or how would you propose validation is done in an acceptable manner?

We have an existing performance test suite, and it should be possible to run that against any new implementation.

[hassila]

Exposing io_uring should probably be done in the context of https://github.com/apple/swift-system ?

It mentions SwiftNIO as one consumer, but doesn't seem like it is currently used - ok to add it as a dependency?

[Lukasa]

It is not ok to add it as a dependency, unfortunately. Until it tags 1.0 adding it as a dependency will immediately cause dependency hell if anyone else in the ecosystem depends on it.

We expose our system calls through our own internal abstraction layer: https://github.com/apple/swift-nio/blob/main/Sources/NIO/Linux.swift.

[hassila]

Ok, that's alright. I'll just make shims and interfaces analogous to what's already in place then.

[hassila]

The inventor of io_uring also provides a convenience library (https://github.com/axboe/liburing) that removes a significant chunk of boilerplate

There's no problem with relying on this library iff its absence can be tolerated on systems that do not have iouring.

Having spent some time now looking at how Swift interacts with C and what support there is in the SPM - I haven't figured out any good way to do that - except manually loading it with dlopen() and using dlsym() to resolve relevant symbols. Is that an acceptable approach (works for me)? I plan on having the implementation fall back to epoll with a warning if io_uring is specified but not available.

[Lukasa]

Yeah, I think we'd have to use dlopen and dlsym. @weissi do you have any objections to that approach?

[weissi]

I'm fine with dlopen/dlsym or invoking the syscalls directly using syscall (returns ENOSYS if unimplemented).

Is the plan to ship the relevant io_uring headers (for struct io_uring_params/struct io_sqring_offsets/struct io_cqring_offsets) in NIO itself?

The other question is if we target liburing or the io_uring_* syscalls directly. Netty targets io_uring_* directly.

[Lukasa]

@weissi See the discussion above. I think targeting liburing is sensible.

[hassila]

Agree, it seems that liburing (also written by Jens Axboe) removes a reasonable amount of boilerplate that otherwise would be reimplemented as discussed above.

With regard to the headers, I had planned on including them such as:
// Pull in io_uring if it's availble (Linux kernel 5.1+ systems, better performance with 5.11+)
#if __has_include(<liburing/io_uring.h>)
#include <liburing/io_uring.h>
#endif

This would require liburing to be installed on a developers build system if liburing is to be used - I did not plan to pull them into NIO. If built without liburing headers, liburing support would not be compiled in and NIO would fall back on using epoll as today.

This would fundamentally bind a compiled binary to have a header version matching the shared library installed for runtime.

I guess the alternative would be to pull in liburing as a git submodule (with the caveat it may only build on newer linux systems... needs to be checked what happens) and effectively bundle both headers and .so - but I think it is probably nicer to try to support the version installed on the production/deployments hosts?

Another alternative would of course be to write all the boilerplate, fundamentally duplicating parts of liburing.

[Lukasa]

I think as a first pass we can require that to get a NIO that supports io_uring you need to build on a system with liburing development headers. If we find that this leads to unsatisfactory outcomes we can always change the behaviour later.

[hassila]

Ok, some initial work done:

• Build with liburing (headers + dynamic loading in place) if available, exposing in CNIOLinux.h (implemented in shims.c)
• Run into slight problem that some of the API of liburing is defined as static inline functions in the header (so cannot dlsym those), but so far all of them seems to resolve down to atomics which should be available at link time and I believe we can call the functions directly without link issues (stubbing those for builds without liburing). If any of those actually require liburing symbols it would push us to require LD_PRELOADING:ng liburing, but that's a problem for another day right now.

Now that the API is accessible a few questions starts to popup, especially I'm trying to figure out the cleanest way to integrate this that would be acceptable.

I would expect that different EventLoops should be able to choose whether to use epoll or liburing? As liburing can optionally be setup with submission queue polling (IORING_SETUP_SQPOLL) through a kernel-side thread (that lives for x ms) for high-throughput scenarios (significantly reducing the need for actual syscalls) - this basically trades of CPU for better latency/throughput. It's also possible to configure both submission/completion queue sizes independently (not uncommon to require a larger completion queue according to presentations from Jens). For the users who want to use io_uring, it seems reasonable to open up for tuning these options on a per-uring basis (we would use one io_uring instance per Eventloop), as one may want to make one uring instance polling while others not.

Any Channel associated with a liburing enabled EventLoop needs to use liburing primitives also for submitting work.

So trying to break down some actual questions:

• Where should I expose configuration of whether to use liburing instead of epoll? SelectableEventLoop extended initialiser? (SelectableEventLoop can then have conditional use of liburing if enabled for the completion handling side)
• If so, should the actual uring options be part of that initialiser also (poll mode, submission/completion queue sizes)?
• I must admit I am a bit unclear of the best way to push in the submission side and would appreciate guidance - should I conditionalize the calls in BSDSocketAPIPosix.swift to use uring instead of syscalls? Then I'd need to add a flag to NIOBSDSocket to know whether to use uring. Not sure yet if/where NIOBSDSocket is set up for the Channel, but if the Channel is associated with an EventLoop with uring enabled it would need to be set for the socket in that case.

Apologies in advance for a bit "fluffy" questions, still wrapping my head around the codebase.

[weissi]

First of all, thank you so much for driving this forward, this is super valuable! Leaving some comments inline.

• Run into slight problem that some of the API of liburing is defined as static inline functions in the header (so cannot dlsym those), but so far all of them seems to resolve down to atomics which should be available at link time

But static inline functions won't necessarily have any symbols at all. They may only exist either as inlined pieces of code. If Swift's clang importer can't import these, then we'd need to "wrap" them in a regular C function and we could stub them out if uring isn't available (exactly as you suggest below).

Now that the API is accessible a few questions starts to popup, especially I'm trying to figure out the cleanest way to integrate this that would be acceptable.

I would expect that different EventLoops should be able to choose whether to use epoll or liburing? As liburing can optionally be setup with submission queue polling (IORING_SETUP_SQPOLL) through a kernel-side thread (that lives for x ms) for high-throughput scenarios (significantly reducing the need for actual syscalls) - this basically trades of CPU for better latency/throughput. It's also possible to configure both submission/completion queue sizes independently (not uncommon to require a larger completion queue according to presentations from Jens). For the users who want to use io_uring, it seems reasonable to open up for tuning these options on a per-uring basis (we would use one io_uring instance per Eventloop), as one may want to make one uring instance polling while others not.

So I'd do the configuration on the EventLoopGroup. If somebody wants N eventloops where say one EventLoop has a different config, then they can just start two EventLoopGroups, one with just 1 EL and another one with N-1. Do you think that'd be good enough?

Any Channel associated with a liburing enabled EventLoop needs to use liburing primitives also for submitting work.

👍

So trying to break down some actual questions:

• Where should I expose configuration of whether to use liburing instead of epoll? SelectableEventLoop extended initialiser? (SelectableEventLoop can then have conditional use of liburing if enabled for the completion handling side)

Good question. I'm not even 100% sure you would even want to use SelectableEventLoop at all. SelectableEventLoop isn't actually public API so we could dynamically use a URingEventLoop or so. I haven't spent a lot of time thinking about this so I think you'd be in a better place to decide.

Aside: There are a few places in NIO itself where we expect SelectableEventLoop but I reckon that could be changed to either expect SelectableEventLoop or URingEventLoop.

• If so, should the actual uring options be part of that initialiser also (poll mode, submission/completion queue sizes)?
• I must admit I am a bit unclear of the best way to push in the submission side and would appreciate guidance - should I conditionalize the calls in BSDSocketAPIPosix.swift to use uring instead of syscalls? Then I'd need to add a flag to NIOBSDSocket to know whether to use uring. Not sure yet if/where NIOBSDSocket is set up for the Channel, but if the Channel is associated with an EventLoop with uring enabled it would need to be set for the socket in that case.

NIOBSDSocket is mostly to name-space lots of socket stuff (like socket options, the right types int vs. SOCKET (Windows) etc) that was introduced to make Windows interop easier. The actual sockets are owned by the Socket class. The instantiation of those is currently done in the bootstraps.

Apologies in advance for a bit "fluffy" questions, still wrapping my head around the codebase.

Not at all, please feel free to ask questions. If you're in some of the Slack/Discord channels, we're also available there for a higher throughput comms channel :)