Add io_uring support to nginx

[CarterLi]

This patch tries to use the new AIO kernel feature io_uring. It simply replace the old Linux AIO code with liburing ( almost function to function change ). Have no time to test it yet.

Any feedback is welcome.

Dependencies

• Linux kernel 5.1 or later
• liburing

More info:

• New kernel polling interface for Linux 4.18 (io_uring)? libuv/libuv#1947
• http://kernel.dk/io_uring.pdf

Detail

Instead of using eventfd to bridge AIO code with epoll module, the patch polls ring_fd directly, which reduces a lot of code ( and may get better performance )

Patch

Should apply to the clean nginx-1.17.0

Patch removed

License

Public domain

[hakasenyang]

Wow! It looks like a good patch.
But I do not think there will be a few people who can use it. (Kernel version problem and no liburing installed)

I'll test this patch and respond again. :)

Can you send me a Pull Request if possible? (excluding .gitignore)

[CarterLi]

But I do not think there will be a few people who can use it. (Kernel version problem and no liburing installed)

I expect liburing will be a standard package for most Linux distributions in the future, just like today's libaio

Can you send me a Pull Request if possible? (excluding .gitignore)

After the patch being stable enough ;)

[CarterLi]

The most efficient way is to replace the whole epoll module with liburing's io_uring_prep_poll_add. I've tried it and failed. The epoll module is too complicate for me :(

[kn007]

conflict with lua nginx module

[kn007]

oh, it's my fault.
when i compile without ccache, it's working fine.

[kn007]

> lsof -n|grep uring
nginx     10174                 root  mem       REG              253,0      31216    3129632 /usr/lib64/liburing.so.1.0.1
nginx     10175                  www  mem       REG              253,0      31216    3129632 /usr/lib64/liburing.so.1.0.1
nginx     10175                  www  mem       REG               0,13                  8801 anon_inode:[io_uring] (stat: No such file or directory)
nginx     10175                  www   35u  a_inode               0,13          0       8801 [io_uring]
nginx     10175 10201            www  mem       REG              253,0      31216    3129632 /usr/lib64/liburing.so.1.0.1
nginx     10175 10201            www  mem       REG               0,13                  8801 anon_inode:[io_uring] (stat: No such file or directory)

hmm

[CarterLi]

I wonder which Linux distro ( uses kernel 5.1 ) are you guys using? :)

[kn007]

> uname -a
Linux kn007.net 5.1.8-1.el7.elrepo.x86_64 #1 SMP Sun Jun 9 08:49:11 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux

yes, 5.1 by elrepo.

which mean the "anon_inode:[io_uring] (stat: No such file or directory)"

[CarterLi]

carter@archlinux-pc ~/n/objs> lsof -n | grep liburing
nginx     3313                carter  mem       REG                8,2     37600       3445 /usr/lib/liburing.so.1.0.1
nginx     3315                carter  mem       REG                8,2     37600       3445 /usr/lib/liburing.so.1.0.1
nginx     3316                carter  mem       REG                8,2     37600       3445 /usr/lib/liburing.so.1.0.1
nginx     3317                carter  mem       REG                8,2     37600       3445 /usr/lib/liburing.so.1.0.1
nginx     3318                carter  mem       REG                8,2     37600       3445 /usr/lib/liburing.so.1.0.1

IDK...

[kn007]

could you using this command?

lsof -n|grep uring

---

你看方便看一下不，取liburing，是看不到提示的。。。
暂时不知道提示有没有什么影响。
想问下，你在nginx上对比过aio和io uring吗？

刚刚测了，快了1.58倍。。（不严谨测试） 实际快了10%+，通过php

[CarterLi]

https://unix.stackexchange.com/questions/463548/what-are-anon-inodes

It's strange that lsof recognizes io_uring fd as a regular file (REG). But because io_uring is not a regular file of course results in No such file or directory

May because lsof doesn't support io_uring well (yet)

[kn007]

OK, thank you.

[CarterLi]

想问下，你在nginx上对比过aio和io uring吗？

No, but there is no doubt that io_uring performs better than Linux AIO. Because Linux AIO only supports files opened with O_DIRECT, other files ( notably buffered files ) are implicitly all handled synchronously.

[CarterLi]

刚刚测了，快了1.58倍。。（不严谨测试）

How did you do that?

[kn007]

@CarterLi
那个太不严谨了，我刚刚编译了两个版本，实测其实差距不大明显。。应该还是我测的方法问题。

[CarterLi]

After 3 hours of debugging, 2 bugs are fixed, patch updated. Still doesn't work though... (it's very strange that ngx_epoll_process_events doesn't seem to be called)

To test it, make sure you have aio on (obviously) and sendfile off because sendfile has higher priority. Note sendfile is useless when using https, because all data need to be encrypted before sending.

[CarterLi]

See #22

[CarterLi]

@denji What did you want to say?