Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

A lightweight single process background worker for Ruby

tree: 510f61bab7

Fetching latest commit…

Cannot retrieve the latest commit at this time

README.md

sqew (pronounced "skew")

sqew is a lightweight background processor. You start a single process that will act as a queue manager and will work multiple jobs concurrently. sqew is short for "small queue" and is not meant to be an all-in-one scalable solution. sqew adopts a common enqueuing API found in other projects to make migrating to other background processors easy.

A skewed photograph by http://www.flickr.com/photos/adriannier/5367065485/

(image by http://www.flickr.com/photos/adriannier/5367065485/ )

When would sqew be a good fit for my project?

  • You don't need to split workers across multiple machines.
  • You don't want to manage multiple background worker processes, but you do want multiple jobs to run concurrently.
  • You don't want to manage an external service like Redis or Mongo to store the queue.
  • You don't want to worry about threading issues.
  • You don't want to worry about long running processes leaking memory.
  • You don't care about the enqueueing or job forking performance.
  • You don't need multiple queues (this may change soon).

If these don't fit the bill or you need more power, I recommend you try the other great gems such as Resque, Sidekiq, and Qu.

Rails Installation

Add this line to your application's Gemfile:

gem sqew, :require "sqew/rails"

Add an initializer in config/initializers/sqew.rb

Sqew.configure do |config|
  config.db = "#{Rails.root}/tmp/"
  config.server = "http://0.0.0.0:8884"
end

The db config will be a directory where sqew will manage its databases, and the server config is what what the worker will connect at as well as where the application will post jobs to.

Once you have sqew configured, you can start the queue manager by running rake sqew:work. This will manage the queues, it'll act as a server where the application post jobs, and it will work the jobs as the arrive.

Installation

Add this line to your application's Gemfile:

gem sqew

And then execute:

$ bundle

Or install it yourself as:

$ gem install sqew

The Sqew Manager

The Sqew manager is a JSON API for inspecting the queue, pushing work onto the queue, and manipulating the queue and workers. Actions you can perform are:

# enqueue a job
# Sqew.push(TheJobClass, 1, 2, 3)
POST /enqueue
  {"job":"TheJobClass", "args":[1, 2, 3]}

# ping the server to programatically see if it is alive
# Sqew.ping
GET /ping

# get the status of the queue, running jobs, failed jobs, and how many workers the server will use
# Sqew.status
GET /status

# dynamically change the number of workers (processes) the manager will use
# Sqew.workers = 10 (the default is 3)
PUT /workers
  "10"

# clear the entire queue
# Sqew.clear
DELETE /clear

# clear just the failed jobs
# Sqew.clear("failed")
DELETE /clear
  failed

# delete a specific job by id
# Sqew.delete(11)
DELETE /11

Enqueuing jobs

From your application you will create jobs just like Resque, Sidekiq, and Qu in the form of

class MyJob
  def perform(arg1, arg2)
    # .. the job code
  end
end

And the manager will receive the job and start working on it when it can. You can enqueue the job from Sqew:

Sqew.push(MyJob, 1, 2)

If you're using Rails 4 you can enqueue the job by using the Rails queuing API:

Rails.queue.push(MyJob, 1, 2)

Using Sqew within Tests

Within your tests, you can set Sqew.inline = true and jobs will work immediately as they are enqueued and errors will be re-raised.

Is it any good?

Yes.

Where does sqew store its data?

Sqew doesn't use redis or your rails database. It uses LevelDB to manage its persistence. So you don't have to manage/setup anything.

Does it work on windows or jruby?

No. It uses [forking](http://en.wikipedia.org/wiki/Fork_(operating_system\)) heavily.

TODO Soon

  • Reusable God/Bluepil/Monit config for managing the worker
  • Capistrano plugin for restarting the worker on deploy
  • Javascript browser front-end for the manager
  • Multiple queues with weight (possibly)

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Added some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request
Something went wrong with that request. Please try again.