-
Notifications
You must be signed in to change notification settings - Fork 366
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
About greenlet worker and OS signals #755
Comments
Let me investigate, thanks for the report. |
I've added a change in 361e380 that seems to address the issue you described, as well as a potential separate issue where a SIGTERM is handled, but does not actually interrupt the running (blocking) task. Do you mind pulling those changes in and taking a look when you have time? |
Thanks for the quick reply! I did some experiments, and I think that those changes solve the problem. In any case, I leave here my findings as it may be helpful to you: 1. SIGTERM:1.1. When no greenlet is executing, the signal is handled by the main thread.In this case, the signal handler will raise a If there are greenlets "waiting" for an I/O result, the consumer will wait on them (via 1.2. When there is a greenlet executing, the signal is handled by the greenlet.In this case, the signal handler stops the execution of the running greenlet, and sets the necessary flags for the consumer to shutdown (not gracefully). If there were no other greenlets ready to be run, the consumer will exit right away. If a greenlet was waiting for the result of an I/O operation, the consumer will also exit right away and that particular result will be lost (which is expected). Nonetheless, Huey's If there were other greenlets waiting to be run, one of them will start executing (despite the consumer already exiting). This will not block the main thread, but the task being run will be lost (no In my case, I'm reenqueuing a task in a 2. SIGINT:2.1. When no greenlet is executing, the signal is handled by the main thread.This case is the same as 2.2. When there is a greenlet executing, the signal is handled by the greenlet.In this case, the greenlet will continue executing (it will not be stopped by the signal). Once it finishes, one of two things can happen depending on whether there is a greenlet ready for execution up next. If that is the case, it will start executing. Otherwise, the consumer will enter graceful shutdown mode, and it will wait for all the greenlets to finish. |
I may need to get a little more clever with this - let me think on it some more and see if I can't get the behavior to more precisely match the threads/processes. |
Awesome! Let me know if I can help with anything. |
I'm not quite sure yet what the best option will be, but this clarifies a bit some of the difficulty:
Looking at the source (and by testing locally), the signal handlers are all run on the gevent Hub. As the comment indicates, we can't switch there. I've tried making a small change here: Testing locally it looks to be working OK. |
The behavior I'm observing with the above change (9cbf17e) is:
I'm not sure we can do a whole lot better than that, but am open to ideas. Closing for now. |
Why was the change in 361e380 done only for I am using |
I don't see this. In the
Here is the code to invoke the task:
As you can see above, the consumer picks up the task and starts executing it. I send a SIGINT to the consumer and the 3 idle workers shutdown. Then once the task has finished it shuts down cleanly and the consumer process exits. |
Sorry, I didn't explain the full story. I am periodically restarting workers by sending If commit 361e380 also included |
Interesting, OK let me take a look. |
Should be fixed here 2e3f08b |
Hi! I'm using the
Greenlet
worker, and noticed that if a signal is received while one of the greenlets is executing (i.e. not "blocked" waiting for I/O), it will be that "thread" the one who handles the signal.This leads to a somehow troublesome behavior. For instance, when a
SIGINT
is received, it can lead to only one greenlet being stopped, instead of the whole event loop (the other greenlets keep going on).I just wanted to know your thoughts on this. Is there a way to force that the signal is handled outside the event loop? Should I even worry about this, as the chances of it happening should be extremely low (tasks are heavy on I/O, not CPU)?
The text was updated successfully, but these errors were encountered: