Reliable queues #43
Reliable queues #43
Conversation
Queues are polled with the atomic RPOPLPUSH command to store the job on an in-flight list. If the job was completed through termination, it is requeued. Otherwise it is just removed from the in-flight list. An additional process can be used to requeue jobs in the case a worker was terminated improperly (e.g. via a SIGKILL).
If end(false) is called on the job executor, the job that is currently running is allowed to finish. Consequently, we should not requeue a job in that case.
|
@danieldk After looking through this I do have one concern, compatibility with Resque and others. Since this pull-request essentially reverses the direction the queues are used, I'm not sure we will be compatible anymore. Do you have any thoughts on this? |
|
I agree. It's a shame that Redis doesn't have lpoprpush, lpoplpush, etc. Do you think that'll better? If so, I'll make a new, rebased pull request On Sat, Mar 1, 2014 at 12:26 AM, Greg Haines notifications@github.comwrote:
|
|
Yes, I like that idea. Maintaining compatibility is important. I look forward to a native lpoprpush. I see the Redis issue and gave it a +1 but I'm not sure it has much traction, unfortunately. |
|
This ticket is superseded by #44. |
This branch implements durable/reliable queues. Queues are polled with the atomic RPOPLPUSH command to store the job on an in-flight list. If the job was completed through termination, it is requeued. Otherwise it is just removed from the in-flight list.
An additional process can be used to requeue jobs in the case a worker was terminated improperly (e.g. via a SIGKILL).