Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job moved from inflight queue to comma-joined queue on worker shutdown in RESET_TO_HIGHEST_PRIORITY strategy. #204

Open
pavan-kalyan opened this issue Feb 18, 2022 · 0 comments

Comments

@pavan-kalyan
Copy link

I have noticed that in certain scenarios (immediate worker shutdown), jobs are removed from the inflight queue and then moved to another queue.

Consider the case where I have the following queues: q1, q2, q3, q4.
My workers are meant to process jobs from these queues.
WorkerPoolImpl::getNextQueue() returns a string of "q1,q2,q3,q4" (for RESET_TO_HIGHEST_PRIORITY) and this is stored in curQueue variable.

During pop, the lua script (fromMultiplePriorityQueues.sha) pops from a single queue and returns the job. But when the worker is shutting down, the jobs are removed from the inflight queue and pushed to a queue "q1,q2,q3,q4". resulting in a new key on redis "namespace:queue:q1,q2,q3,q4". This code is in WorkerPoolImpl::removeInFlight.

What is the purpose of storing the jobs in this comma-joined queue? From what I could see, no worker would poll from this queue as the lua script would always pop from the individual queues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant