Redis queue driver may never pop delayed jobs if the push rate is high

[halaei]

Laravel Redis driver uses check-and-set Redis transactions (implemented with Redis Watch, Multi and Exec commands) to move the expired jobs from delayed and reserved scheduled queues into the FIFO queue. I think using the check-and-set transaction is not a good idea and it should be replaced with Exec transactions (using Lua scripts). My reason is that, when the push rate into delayed queues are high and the queue get large, check-and-set transactions will fail with very high probability. On the other side, as far as I know, using the Exec transactions may only lead to a tolerable delay on push operations.

I will try to complete my argument and give pull requests if necessary.

Related issues:

1. [5.1] [5.2] Multiple queue workers using Redis: [5.1] [5.2] Multiple queue workers using Redis #8577
2. Uncaught Predis exception Predis\Transaction\AbortedMultiExecException: Uncaught Predis exception Predis\Transaction\AbortedMultiExecException #8576
3. RedisQueue should use transactions for atomicity and avoiding race conditions: RedisQueue should use transactions for atomicity and avoiding race conditions #4912
4. RedisQueue::pop() has 2 issues: RedisQueue::pop() has 2 issues #6366

[halaei]

Here is a benchmark
halaei/redis-benchmark