Add priority reliable queue #4
Conversation
So that a single worker can process work from multiple queues of different priority (e.g. a "critical" queue, a default queue and a "backlog" queue), add a new Priority Reliable Queue class which will check a given array of Redis keys for work in a specific order. This will ensure that the highest priority queue is empty before checking the next queue for work and so on until the last queue.
|
An issue with this is that it relies on polling all queues with See redis/redis#1785 and sidekiq/sidekiq#1769 for the discussion of a new command that would make pop-push-ing multiple queues easier. |
src/PriorityReliableQueue.php
Outdated
| } | ||
| } | ||
|
|
||
| public function rewind() |
There was a problem hiding this comment.
I'm not sure I understand what is the meaning of rewind in the context of the queue? Is it only here to make sure we conform to the Iterator interface?
There was a problem hiding this comment.
Yes, take a look at the Iterator interface, it's a key part of how foreach works:
In short, when you loop over an iterator, it calls the following methods in order:
rewindvalidcurrentkeynext- Go to 2.
There was a problem hiding this comment.
I guess I'm just not used to rewind being stateful, but maybe that's okay for practical reasons 🤔
|
See Sidekiq's Weighted random algorithm for an alternate algorithm we could try here. |
|
@mudge why is this better than having processes with different priorities? |
|
@AnnaKL it's not necessarily "better": it would be an alternative to the current solution of running one worker per queue. The idea here being that you could run fewer, simpler workers (e.g. all configured identically) and have them consume and process work of different priorities without having to scale up and down yourself. At the moment, if you want to consume a |
mudge
force-pushed
the
priority-queue
branch
3 times, most recently
from
abbeea1
to
b0a7975
Compare
|
As testing randomness across PHP versions is proving somewhat tricky due to backwards-incompatible changes to |
mudge
force-pushed
the
priority-queue
branch
2 times, most recently
from
1acc589
to
f2b475a
Compare
Instead of waiting for each queue to drain in priority order, rely on probability to pull from ordered queues in rough priority. This way, we can block for a second when waiting for work to alleviate pressure on Redis when queues are empty and we can avoid queue starvation by checking all queues even if the highest priority is always full.
So that a single worker can process work from multiple queues of different priority (e.g. a "critical" queue, a default queue and a "backlog" queue), add a new Priority Reliable Queue class which will check a given array of Redis keys for work in a specific order.
This will ensure that the highest priority queue is empty before checking the next queue for work and so on until the last queue.