Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Browse files

Update README.

  • Loading branch information...
commit 182f69f75aa0162207344c9beac90b84476fbf19 1 parent 0e16749
@soveran authored
Showing with 82 additions and 35 deletions.
  1. +82 −35 README.md
View
117 README.md
@@ -8,76 +8,123 @@ Redis based queues and workers.
Description
-----------
-**Ost** makes it easy to enqueue object ids and process them with workers.
+**Ost** makes it easy to enqueue object ids and process them with
+workers.
-Say you want to process video uploads. In your application you will have something like this:
+Say you want to process video uploads. In your application you will
+have something like this:
- Ost[:videos_to_process].push(@video.id)
+``` ruby
+Ost[:videos_to_process].push(@video.id)
+```
Then, you will have a worker that will look like this:
- require "ost"
+``` ruby
+require "ost"
- Ost[:videos_to_process].each do |id|
- # Do something with it!
- end
+Ost[:videos_to_process].each do |id|
+ # Do something with it!
+end
+```
Usage
-----
-**Ost** connects to Redis automatically with the default options (localhost:6379, database 0).
+**Ost** connects to Redis automatically with the default options
+(localhost:6379, database 0).
You can customize the connection by calling `connect`:
- Ost.connect port: 6380, db: 2
+``` ruby
+Ost.connect port: 6380, db: 2
+```
Then you only need to refer to a queue for it to pop into existence:
- Ost[:rss_feeds] << @feed.id
+``` ruby
+Ost[:rss_feeds] << @feed.id
+```
A worker is a Ruby file with this basic code:
- require "ost"
+``` ruby
+require "ost"
- Ost[:rss_feeds].each do |id|
- ...
- end
+Ost[:rss_feeds].each do |id|
+ # ...
+end
+```
-It will pop items from the queue with a timeout of two seconds and retry indefinitely. If you want to configure the timeout, set the environment variable `OST_TIMEOUT`.
+It will pop items from the queue as soon as they become available. It
+uses BRPOPLPUSH with a timeout that can be specified with the
+`OST_TIMEOUT` environment variable.
-Available methods for a queue are `push` (aliased to `<<`) and `pop` (aliased to `each`).
+Note that in these examples we are pushing numbers to the queue. As
+we have unlimited queues, each queue should be specialized and the
+workers must be smart enough to know what to do with the numbers they
+pop.
-Note that in these examples we are pushing numbers to the queue. As we have unlimited queues, each queue should be specialized and the workers must be smart enough to know what to do with the numbers they pop.
+Available methods
+=================
+
+`Ost.connect`: configure the connection to Redis. By default, it
+connects to localhost in port 6379 and uses the database 0. It accepts
+the same options as [redis-rb](https://github.com/redis/redis-rb).
+
+`Ost.stop`: halt processing for all queues.
+
+`Ost[:example].push item`, `Ost[:some_queue] << item`: add `item` to
+the `:example` queue.
+
+`Ost[:example].push { |item| ... }`, `Ost[:example].each { |item| ...
+}`: consume `item` from the `:example` queue. If the block doesn't
+complete successfully, the item will be left at a backup queue.
+
+`Ost[:example].stop`: halt processing for the `example` queue.
Failures
========
-If the block raises an error, it is captured by **Ost** and the exception is logged in Redis.
-
-Consider this example:
+**Ost** stores in-process items in backup queues. That allows the
+developer to deal with exceptions in a way that results adequate
+for his application.
- require "ost"
+There is one backup queue for each worker, with the following
+convention for naming the key in Redis: given a worker using the
+`:events` queue, running in the hostname `domU-12-31-39-04-49-C7`
+with the process id `28431`, the key for the backup queue will be
+`ost:events:domU-12-31-39-04-49-C7:28431`.
- Ost[:rss_feeds].each do |id|
- ...
- raise "Invalid format"
- end
+Here's the explanation for each part:
-Then, in the console you can do:
+* `ost`: namespace for all **Ost** related keys.
+* `events`: name of the queue.
+* `domU-12-31-39-04-49-C7`: hostname of the worker.
+* `28431`: process id of the worker.
- >> Ost[:rss_feeds].push 1
- => 1
+Priorities
+----------
- >> Ost[:rss_feeds].errors
- => ["2010-04-12 21:57:23 -0300 ost:rss_feeds:1 => #<RuntimeError: Invalid format>"]
+There's no concept of priorities, as each queue is specialized and you
+can create as many as you want. For example, nothing prevents the
+creation of the `:example_high_priority` or the
+`:example_low_priority` queues.
Differences with Delayed::Job and Resque
---------------------------------------
+----------------------------------------
-Both [Delayed::Job](http://github.com/tobi/delayed_job) and [Resque](http://github.com/defunkt/resque)
-provide queues and workers (the latter using Redis). They provide dumb workers that process jobs, which are specialized for each task. The specialization takes place in the application side, and the job is serialized and pushed into a queue.
+Both [Delayed::Job](http://github.com/tobi/delayed_job) and
+[Resque](http://github.com/defunkt/resque) provide queues and workers
+(the latter using Redis). They provide dumb workers that process jobs,
+which are specialized for each task. The specialization takes place
+in the application side, and the job is serialized and pushed into a
+queue.
-**Ost**, by contrast, just pushes numbers into specialized queues, and uses workers that are subscribed to specific queues and know what to do with the items they get. The total sum of logic is almost the same.
+**Ost**, by contrast, just pushes numbers into specialized queues, and
+uses workers that are subscribed to specific queues and know what to
+do with the items they get. The total sum of logic is about the same,
+but there's less communication and less data transfer with **Ost**.
Installation
------------
@@ -87,7 +134,7 @@ Installation
License
-------
-Copyright (c) 2010 Michel Martens
+Copyright (c) 2010, 2011, 2012 Michel Martens
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
Please sign in to comment.
Something went wrong with that request. Please try again.