Asynchronous remove_writer and remove_reader. #61
Comments
|
I'm not sure I understand why this 100% CPU loop happens. From the libuv docs:
It seems that if you call
I'd really appreciate if you could share your view on this. |
|
Also, we can probably use |
|
Hi Yury, I believe the CPU is being burned in the libuv poll here, https://github.com/libuv/libuv/blob/v1.x/src/unix/linux-core.c#L327-L348 That in itself may be a libuv bug but I have less experience with their community to say for certain. From what I've gathered epoll is susceptible to confusion when you feed it a dup() FD, then close the dup() based FD but leave the original file description open. As far as epoll is concerned the file description (not descriptor) pointed to by the dup() based FD is still open, so it may trigger events for an FD that is no longer valid; Attempts to use EPOLL_CTL_DEL on that FD will fail as you can see in the strace output. Some context from epoll(7)..
Regarding calling The python code that produced this result is from aiohttp and the only setup required was to serve a single, but large, file over a link that will cause os.sendfile to raise BlockingIOError. https://github.com/KeepSafe/aiohttp/blob/master/aiohttp/file_sender.py#L103-L110 As you can see, that code will always call |
|
Here is a simplified test that will reproduce the issue. import os
import uvloop
loop = uvloop.EventLoopPolicy().get_event_loop()
rfd, wfd = os.pipe()
wfd2 = os.dup(wfd)
def on_write_event():
loop.remove_writer(wfd2)
os.close(wfd2)
loop.add_writer(wfd2, on_write_event)
loop.run_forever() |
mayfield
changed the title
|
Seems that we can manually call @saghul Is this something we should fix in libuv somehow? |
|
My test cases are passing now. Thank you very much @1st1! |
|
BTW, do you have any idea how to write a unittest for this in uvloop? |
|
I've been just using time.process_time() to check CPU usage after letting the loop run for a second. I wouldn't be comfortable putting that into a unittest but here it is for reference.. import asyncio
import os
import time
import uvloop
loop = uvloop.EventLoopPolicy().get_event_loop()
rfd, wfd = os.pipe()
wfd2 = os.dup(wfd)
def on_write_event():
print(" START REMOVE")
loop.remove_writer(wfd2)
print(" DONE REMOVE - START CLOSE")
os.close(wfd2)
print(" DONE CLOSE")
loop.add_writer(wfd2, on_write_event)
ct = time.process_time()
loop.run_until_complete(asyncio.sleep(0.100, loop=loop))
loop_cputime = time.process_time() - ct
assert loop_cputime < 0.050, 'libuv is running hot!' |
Yeah, process_time isn't going to be reliable. I've added a direct regression test of epoll/fd. Will make a release shortly. |
|
Closing this one, should be fixed in |
|
Is this fix added in C library of libuv as well? |
I notice that
remove_readerandremove_writerare asynchronous with respect to when the file descriptor is actually removed from its epoll set (linux terminology). This is a byproduct of how libuv works.This is different from the asyncio module and causes troubles with at least dup() based FDs. Namely serving static files with aiohttp under uvloop will induce a 100% CPU loop inside libuv because it can't remove the FD. Strace attached..
I could go into more detail about this but it's rather complex and I believe there are a number of issues you could tie back to the fact that remove_* isn't syncronous as it is with the stdlib implementation of EventLoop.
The text was updated successfully, but these errors were encountered: