Skip to content

Latest commit

 

History

History
141 lines (100 loc) · 4.95 KB

README.md

File metadata and controls

141 lines (100 loc) · 4.95 KB
Raw

Redis Queue Battles Message Groups!

Fairy is a lightweight queue engine for node.js based on Redis. Fairy offers ActiveMQ's message groups alike feature which can guarantee the sequential processing order of tasks belong to a same group.

But, unkile message groups, Fairy doesn't always route tasks of a group to a same worker, which can lead to unwanted waiting time when:

  1. Tasks of group X and Y are appointed to worker A.
  2. Worker A is processing tasks of group X sequentially.
  3. Tasks of group Y are pending, while:
  4. Worker B is still idling because of 1.

Fairy will route the task of group Y to worker B in this scenario.

Fairy takes a different approach than Message Groups. Instead of making all tasks of a same group be routed to the same consumer, Fairy route a task to any worker when there's no processing tasks of the same group.

The design philosophy makes Fairy ideal for the following requirements:

  1. Tasks of a same groups need be processed in sequence.
  2. Each worker processes tasks in serial.

Fairy takes a different approach than Message Groups. Instead of making all tasks of a same group be routed to the same consumer, Fairy route a task to any worker when there's no processing tasks of the same group.

Resque cannot guarantee the processing order of the tasks although the task queue is FIFO. The more workers you have, the more possible you'll encountering concurrency which breaks the processing order of tasks in the same group.

Installation

npm install fairy

Get Started

The minimium set of APIs you need to learn in order to implement a task queue system are:

  • enqueue tasks, and
  • regist a function for processing them.

Enqueue Tasks

Provide as many parameters as you want (and an optional callback function). The first argument will be used for message grouping.

queue = require('fairy').connect().queue('task_name')
queue.enqueue 'foo', 'bar', ->
  console.log 'your order has been placed, sir.'

Register Task Handler

When registered a task handler, the Fairy queue becomes a worker automatically.

The registered handler function will be called when there're tasks to be processed, with the enqueued parameters of the task. The last argument will be a non-optional callback function.

Arguments of the callback function follow node.js error handling convention: err and res.

Calling the callback function is your responsibility (or Fairy will not dispatch tasks to the worker and block tasks of the same group forever!)

queue = require('fairy').connect().queue('task_name')
queue.regist (param1, param2, callback) ->
  # Do your work here, be it synchronous or asynchronous.
  callback err, res

Web Front-End

Fairy comes with a ready-to-use web front-end. Simply insert the middleware into the pipeline:

app = require('express').createServer()
fairy_web = require 'fairy/web'
app.use fairy_web.connect().middleware
app.listen 3000

More APIs

More APIs include:

  1. Objects of Class Queue:
  • Placing tasks -- enqueue
  • Regist handlers -- regist
  • Reschedule tasks -- reschedule
  • Query status --
    • recently_finished_tasks
    • failed_tasks
    • blocked_groups
    • slowest_tasks
    • processing_tasks
    • workers
    • statistics, etc.
  1. Objects of Class Fairy:
  • queues, return all queues.
  • statistics, return statistics of all queues.

See Example Folder for demos. Or review the Annotated Source Code for complete API explanations.

License

Copyright (c) 2012 Baoshan Sheng

Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following
conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.