README.md

RedisCall

A Redis access library, which provides:

• Connection management
  • Per-thread connection
  • Shared between threads connection
• Key names construction
• Transactions
  • Queued result handling
  • Retry on optimistic lock fail
• Queues
  • message handling with backup queue
  • restore from backup queue
  • queues in Rails controllers
  • messages as REST resources
  • message handling in separate process
  • graceful stop of blocking operations
• JSON encode/decode
• And nice ruby API

It use Hiredis to connect to Redis.

Basic operations

r = RedisCall.new

prefix = r.key('foo')

# set the 'foo.bar.qux' to 'content' with 1 day expiration
r.setex prefix+:bar+'qux', 1.day, 'content'
r.setex prefix/:bar/'qux', 1.day, 'content'

# get 'foo.bar.qux'
r.get prefix/:bar/'qux'

Custom storage class

class MyStorage < RedisCall
  def store key_name, value
    multi do
      del key(:prefix)/key_name/:some_state
      set key(:prefix)/key_name, value
    end
  end
end

Queues

Basic operations

queue = RedisQueue::Base.new('foo')
queue.push(hello: 'darling')

queue.name
queue.length
queue.backup_length

queue.pop_all
queue.backup_elements
queue.backed_up_pop_all

subqueue = RedisQueue::Base.new(queue.name/:subqueue)

Message handler process

Here is sample standalone message handler process with backup queue. To handle process-related functionality (start/stop/watchdog), workety library is used.

class Handler
  def start
    @queue = RedisQueue::Base.new 'queue_name', connect: true
    @queue.restore_backup

    @handler = Thread.networkety do
      loop do
        @queue.backed_up_blocking_pop do |message|
          puts message.inspect
        end
      end
    end
  end

  def join
    @handler.log_join "Handler thread joined"
  end

  def stop
    @queue.disconnect @handler
  end
end

Serialization and custom queue class

Redis operates with string as a queue element. To transfer Ruby classes, serialization technique may be used. RedisCall use JSON as the transfer format. In case when you want to transfer not Hash but some custom class (say MyElement), you may define your own queue class to handle that:

class MyQueue < RedisQueue::Base
  def encode element
    super(element.as_json(:except => [:serialized]))
  end

  def decode raw
    MyElement.new(super(raw))
  end
end

Queue metadata

You may store queue metadata in config/redis_queue.yml file:

queue_name:
  attr: value

and retrieve it:

RedisQueue::Base.new('queue_name').config[:attr]
# => 'value'

Queues in Rails controllers

A list of queues

class QueueListController
  # Queue names currently are not checked for funny characters by library,
  # so a potential of Redis command injection exists.
  def filtered_id
    params[:id].tr('^A-Za-z0-9.', '')
  end

  def index
    @queues = RedisQueue::Base.all
  end

  def destroy
    @queue = RedisQueue::Base.find filtered_id
    @queue.destroy
  end
end

Messages

class MessagesController
  def queue
    @queue ||= RedisQueue::Base.new('input')
  end

  def output_queue
    @output_queue ||= RedisQueue::Base.new('output')
  end

  def raw_message
    params[:message][:serialized]
  end

  def index
    @messages = queue.elements
    @messages = queue.backup_elements

    @messages = queue.pop_all
    @messages = queue.backed_up_pop_all
  end

  def update
    queue.insist do # Retry on optimistic lock fail
      queue.watch_backup # Optimistic locking

      if queue.raw_backup_elements.include? raw_message
        @message = queue.decode raw_message

        if @message.valid?

          queue.multi do
            queue.remove_raw_backup_element raw_message
            output_queue.push "some message"
          end

          respond_to do |format|
            format.html { render :nothing => true, status: :created }
          end
        else
          respond_to do |format|
            format.html { render partial: 'form', :locals => {message: @message}, status: :unprocessable_entity }
          end
        end
      else
        respond_to do |format|
          format.html { render text: "The message you are trying to update is gone", status: :gone }
        end
      end
    end
  end

  def destroy
    queue.remove_raw_element raw_message
  end
end

Environment details

Connection configuration

By default, Redis is expected to be found listening the 127.0.0.1:6379, but that can be configured in config/redis_call.yml

development:
  host: localhost
  port: 6379

production:
  host: localhost
  port: 6379

test:
  host: localhost
  port: 6379

Forks and Unicorn webserver

Forking the process require reconnection. For Unicorn webserver it may be done by stating in config/unicorn.rb:

before_fork do |server, worker|
  RedisCall.new.disconnect
end

after_fork do |server, worker|
  RedisCall.new.connect
end

Copyright and License

Copyright 2011 Stanislav Senotrusov

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Contributing

Please see our Contributing Guidelines for details.