(BL)POP considered dangerous

[dmag]

There's a pretty big window of opportunity for failure after the (BL)POP, but before processing is completely finished. Any problem and the job will be lost forever. (Problems could be kill -9, power loss, out of memory during fork, kernel OOM killer, etc.)

Redis has the RPOPLPUSH command to fix this exact problem (but it requires a bit more complexity, such as per-worker queues or other timeout mechanism). See "Programming patterns: safe queues"
http://code.google.com/p/redis/wiki/RpoplpushCommand

I guess the big question is: Should safety be a first class property of Resque, or should it be relegated to a plug-in? (At the very least, this issue should be documented in the meantime).

[defunkt]

Resque does not advertise itself as a system that will never lose your jobs. Part of the design is we don't care if jobs are lost.

If this is important to you, there are plenty of systems which can offer that feature.

Also: as long as your machine doesn't go away forever (e.g. EC2 instance), Resque won't lose jobs due to kill -9, power loss, out of memory during fork, kernel OOM killer.