Add io_uring support to nginx (Fixes #21) #22
Conversation
|
Merged. Thanks! |
|
@CarterLi have you tested this with elrepo yum linux 5.2.5 kernels ? i tried this with centos 7.6 with elrepo 5.2.5 kernel and get the following |
|
ah i forgot to install liburing ! |
How do i fix it? |
|
How can I reproduce it? @kn007 |
|
I don't know... nginx ENV: nginx 1.17.3 with my nginx patch, nginx_io_uring, use_openssl_md5_sha1 patches. Hugepages enable for mysql group |
|
@kn007 Please test if reducing the number |
|
Thanks @CarterLi , I will try it now. reduced it to 32, failed. My kvm server SCSI Controller is Created a new kvm server with I will rebuild nginx on the new kvm server without another nginx module for test Still failed....... |
|
@CarterLi If I compile the nginx with Check that SELINUX was disabled Found problem... If I build nginx as user so maybe because systemd permisson problem here, i will google for it. using back upstart mode to start nginx... I do not know how to fix this on systemd... |
Thank you for your feedback. I will investigate it when I have time
io_uring is very new, so there are very few resources to search. |
|
I can't reproduce this issue on Ubuntu Look for |
|
|
|
@CarterLi It seems that CentOS 8 will be released soon, I hope to solve this problem indirectly. |
|
Found this: https://github.com/torvalds/linux/blob/master/fs/io_uring.c#L2767 Please try to raise the size of |
|
ulimit -l i will try to set this on systemd script |
Strange. Can you please get the size of Wait, please |
|
Bingo. On my system (Arch) the default is Another issue Iʼm having now is that Iʼm getting a lot of things like Sometimes things like this This is not normal, right? |
It happened to me a few times too. I guess that it may relate to this kernel issue Let's see if Linux 5.3 will fix this issue.
Oh yes, I found it in the error log too, but I can't reproduce it consistently. Needs more investigation.
|
|
@CarterLi After Now I am add Something useful for this: https://serverfault.com/a/678861
Nginx limit status |
|
@CarterLi
echo 3 > /proc/sys/vm/drop_caches && swapoff -a && swapon -aBad address issue occurred when nginx was restarted and the file was tried to access after emptying PageCache etc. I think we should use functions such as io_urging_setup, io_urging_register, etc. Reference : liburing/test/io_uring_register.c int
new_io_uring(int entries, struct io_uring_params *p)
{
int fd;
fd = io_uring_setup(entries, p);
if (fd < 0) {
perror("io_uring_setup");
exit(1);
}
return fd;
}
// ...
while (nr_fds) {
ret = io_uring_register(uring_fd, IORING_REGISTER_FILES,
fd_as, nr_fds);
if (ret != 0) {
nr_fds /= 2;
continue;
}
// ... |
Strange. I can reproduce this issue on nginx but I can't reproduce it separately. #include <liburing.h>
#include <libaio.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
char str[4096];
void test_uring(int sfd) {
struct io_uring ring;
io_uring_queue_init(32, &ring, 0);
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
struct iovec iov = {
.iov_base = str,
.iov_len = sizeof(str),
};
io_uring_prep_readv(sqe, sfd, &iov, 1, 0);
io_uring_submit(&ring);
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);
io_uring_cqe_seen(&ring, cqe);
io_uring_queue_exit(&ring);
}
void test_aio(int sfd) {
io_context_t context;
io_queue_init(32, &context);
struct iocb iocb, *piocb = &iocb;
io_prep_pread(piocb, sfd, str, sizeof(str), 0);
io_submit(context, 1, &piocb);
struct io_event event;
io_getevents(context, 1, 1, &event, NULL);
io_queue_release(context);
}
void test_old(int sfd) {
read(sfd, str, sizeof(str));
}
int main() {
int sfd = open(__FILE__, O_RDONLY);
test_uring(sfd);
// test_aio(sfd);
// test_old(sfd);
close(sfd);
puts(str);
}I guess there are other unknown requirements.
I guess no. According to the manpage, files that can be registered by BTW, there are wrapper functions for |
|
Do I need to patch the patch file? When i try to patch it with command |
|
https://raw.githubusercontent.com/hakasenyang/openssl-patch/master/nginx_io_uring.patch |
|
Hi, @CarterLi . Any updates for this patch? |
|
No updates. |
|
Because of nginx support of kTLS, this patch is much less valuable. |
Finally having time to fix the old patch, so this is PR
This patch tries to use the new AIO kernel feature
io_uring. It simply replace the oldLinux AIOcode withliburing( almost function to function change ).Any feedback is welcome.
Dependencies
More info:
Detail
Instead of using eventfd to bridge AIO code with epoll module, the patch polls ring_fd directly, which reduces a lot of code ( and may get better performance )
Patch
Should apply to the clean nginx-1.17.0
Use
Compile nginx with
--with-file-aio, and haveaio oninnginx.conf. Additionally setsendfile offbecause sendfile has higher priority. Note sendfile is useless when using https, because all data need to be encrypted before sending.Note io_uring AIO has no known limitation like aio. Buffered I/IO won't block, thus
directiois not necessary.Read more on offical doc: http://nginx.org/en/docs/http/ngx_http_core_module.html#aio
To debug it, first compile nginx with
--with-debug --with-cc-opt='-O0 -g', and setmaster_process offdaemon offerror_log stderr debuginnginx.confLicense
Public domain