use worker_init/worker_process_init to start event processor threads in celery #444
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…in celery Celery uses a prefork model for its worker processes. The app is loaded in a "worker", which then forks one or more "worker processes". Similiarly to the uwsgi prefork issues we had to work around, this fork happens _after_ we created the event processor thread, so the worker processes don't have an event processor thread of their own, and just fill up the event queue without it ever being processed. To work around this, we use worker_init and worker_process_init signals to start the event processor thread _after_ the fork. This only became an issue with version 4.2.0 of the agent. In earlier versions of the agent, threads were started relatively late, after celery already forked the worker processes.
beniwohli
force-pushed
the
celery-prefork-issues
branch
from
cac6cab
to
6b1bd42
Compare
axw
approved these changes
Apr 2, 2019
Not yet, good point. These signals are only fired for the pre-fork worker model AFAIK, so at least it shouldn't make things worse for the other worker pool types. |
|
Update: solo works, eventlet and gevent are missing spans, but this was always the case, as we don't currently support eventlet/gevent. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Celery uses a prefork model for its worker processes. The app is loaded
in a "worker", which then forks one or more "worker processes". Similiarly
to the uwsgi prefork issues we had to work around, this fork happens
after we created the event processor thread, so the worker processes
don't have an event processor thread of their own, and just fill up the
event queue without it ever being processed.
To work around this, we use worker_init and worker_process_init signals
to start the event processor thread after the fork.
This only became an issue with version 4.2.0 of the agent. In earlier
versions of the agent, threads were started relatively late, after
celery already forked the worker processes.