Redis queue

A lightweight Redis-based queue with message acknowledgement and support for multiple consumers and producers.

Installation

gem install redis_queue

Usage

A message lifecycle is as follows:

• Message is pushed to the queue
• Message is popped out of the queue and marked as in use so that no other consumer takes it
• Once the message is processed, the consumer acknowledges the message by marking it as finished or failed or by rolling it back to the queue.

Whenever a consumer dies, its messages won't be processed by another consumer. They are kept inside the queue (in a different Redis key). Calling RedisQueue#reset puts all messages in use back into the queue.

Producer-consumer

Let's build a simple producer that enqueues some messages followed by a consumer that properly processes one message, marks one message as failed, forgets to mark another message as either finished or failed, returns a fourth, and repushes, unpushes and forgets other messages:

require 'redis_queue'

queue = RedisQueue.new
queue.clear

p queue.pop(block: false) # Non-blocking pop, which will return nil when empty

queue.push "message 2"
queue.push "message 3"
queue.push "message 4"
queue.push "message 5"
queue.push "message 6"
queue.push "message 6.5"
queue.push "message 7"
queue.push "message 1", true # This gives priority to this message
queue.remove "message 6.5"

queue.pop.tap do |msg|
  queue.done  msg
  puts        msg
end

queue.pop.tap do |msg|
  queue.fail  msg
  puts        msg
end

queue.pop.tap do |msg|
  puts        msg
end

queue.pop.tap do |msg|
  queue.repush msg
  puts        msg
end

queue.pop.tap do |msg|
  queue.repush msg
  puts        msg
end

queue.pop.tap do |msg|
  queue.forget msg
  puts        msg
end

queue.pop.tap do |msg|
  queue.unpop msg
  puts        msg
end

queue.touch.tap do |msg|
  puts        msg
end

queue.print_stats
queue.print_contents

The output will be:

nil
message 1
message 2
message 3
message 4
message 5
message 6
message 7
message 7
messages enqueued: 3
messages in use:   1
messages failed:   1
messages done:     1
messages enqueued: ["message 4", "message 5", "message 7"]
messages in use:   {"message 3"=>"1521025376623"}
messages failed:   {"message 2"=>"1521025376616"}
messages done:     {"message 1"=>"1521025376603"}

In use, failed, and done lists are Redis hashes which store the messages timestamps in milliseconds.

Copyright

Copyright (c) 2014. MIT license, José Ignacio Fernández <joseignacio.fernandez@gmail.com>