Should rate-limited tasks increment the prefetch limit? #7803
RJPercival
started this conversation in
General
Replies: 4 comments 9 replies
-
can you share a proof of concept of what you are thinking? with a failing test? |
Beta Was this translation helpful? Give feedback.
7 replies
-
@RJPercival can you please try incrementing this and see what happens? |
Beta Was this translation helpful? Give feedback.
1 reply
-
Related issue: #8119 |
Beta Was this translation helpful? Give feedback.
0 replies
-
Has there been any progress on this? |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
When Celery is configured with a prefetch limit of 1, and receives a task with an ETA but that ETA is in the future, Celery will increment the prefetch limit and schedule that task to be run when its ETA elapses. The incremented prefetch limit allows the worker to fetch another task while holding the future-dated task. This seems sensible.
The behaviour is different for rate-limited tasks. If a worker receives a task but there are not sufficient tokens (due to rate-limiting) to execute that task, it will similarly schedule that task to be run when it is expected that a token will be available, but the prefetch limit is not incremented. This means the worker is unable to fetch any more tasks and sits idle. It does so even though the next task it could fetch might not be rate-limited. This seems like a bug - should the prefetch limit not also be incremented in this scenario?
Here is the relevant code:
celery/celery/worker/strategy.py
Lines 190 to 202 in febe695
Beta Was this translation helpful? Give feedback.
All reactions