Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 147 lines (102 sloc) 3.891 kb
d71040e @soveran Initial import.
authored
1 Ost
2 ===
3
a3bdaea @soveran Change name of timeout env variable and add documentation.
authored
4 Redis based queues and workers.
d71040e @soveran Initial import.
authored
5
e88475d @soveran Add picture of Ost Cafe to README.
authored
6 ![Ost Cafe, by Arancia Project](http://farm4.static.flickr.com/3255/3161710005_36566b8a9e.jpg)
7
d71040e @soveran Initial import.
authored
8 Description
9 -----------
10
182f69f @soveran Update README.
authored
11 **Ost** makes it easy to enqueue object ids and process them with
12 workers.
d71040e @soveran Initial import.
authored
13
182f69f @soveran Update README.
authored
14 Say you want to process video uploads. In your application you will
15 have something like this:
d71040e @soveran Initial import.
authored
16
182f69f @soveran Update README.
authored
17 ``` ruby
18 Ost[:videos_to_process].push(@video.id)
19 ```
d71040e @soveran Initial import.
authored
20
21 Then, you will have a worker that will look like this:
22
182f69f @soveran Update README.
authored
23 ``` ruby
24 require "ost"
d71040e @soveran Initial import.
authored
25
182f69f @soveran Update README.
authored
26 Ost[:videos_to_process].each do |id|
27 # Do something with it!
28 end
29 ```
d71040e @soveran Initial import.
authored
30
31 Usage
32 -----
33
0ccf89b @frodsan Update documentation.
frodsan authored
34 Ost uses a lightweight Redis client called [Redic][redic]. To connect to
35 a Redis database, you will need to set an instance of `Redic`, with a URL
36 of the form `redis://:<passwd>@<host>:<port>/<db>`.
d71040e @soveran Initial import.
authored
37
0ccf89b @frodsan Update documentation.
frodsan authored
38 You can customize the connection by calling `Ost.redis=`:
d71040e @soveran Initial import.
authored
39
182f69f @soveran Update README.
authored
40 ``` ruby
0ccf89b @frodsan Update documentation.
frodsan authored
41 require "ost"
42
43 Ost.redis = Redic.new("redis://127.0.0.1:6379")
182f69f @soveran Update README.
authored
44 ```
d71040e @soveran Initial import.
authored
45
46 Then you only need to refer to a queue for it to pop into existence:
47
182f69f @soveran Update README.
authored
48 ``` ruby
0ccf89b @frodsan Update documentation.
frodsan authored
49 require "ost"
50
51 Ost.redis = Redic.new("redis://127.0.0.1:6379")
52
53 Ost[:rss_feeds] << @feed.id
54 ```
55
56 Ost defaults to a Redic connection to `redis://127.0.0.1:6379`. The example
57 above could be rewritten as:
58
59 ``` ruby
60 require "ost"
61
182f69f @soveran Update README.
authored
62 Ost[:rss_feeds] << @feed.id
63 ```
d71040e @soveran Initial import.
authored
64
65 A worker is a Ruby file with this basic code:
66
182f69f @soveran Update README.
authored
67 ``` ruby
68 require "ost"
d71040e @soveran Initial import.
authored
69
182f69f @soveran Update README.
authored
70 Ost[:rss_feeds].each do |id|
71 # ...
72 end
73 ```
d71040e @soveran Initial import.
authored
74
182f69f @soveran Update README.
authored
75 It will pop items from the queue as soon as they become available. It
0ccf89b @frodsan Update documentation.
frodsan authored
76 uses `BRPOPLPUSH` with a timeout that can be specified with the
182f69f @soveran Update README.
authored
77 `OST_TIMEOUT` environment variable.
a3bdaea @soveran Change name of timeout env variable and add documentation.
authored
78
182f69f @soveran Update README.
authored
79 Note that in these examples we are pushing numbers to the queue. As
80 we have unlimited queues, each queue should be specialized and the
81 workers must be smart enough to know what to do with the numbers they
82 pop.
d71040e @soveran Initial import.
authored
83
182f69f @soveran Update README.
authored
84 Available methods
85 =================
86
87 `Ost[:example].push item`, `Ost[:some_queue] << item`: add `item` to
88 the `:example` queue.
89
c6f17cf @matflores Fix pop/each usage docs in README
matflores authored
90 `Ost[:example].pop { |item| ... }`, `Ost[:example].each { |item| ...
182f69f @soveran Update README.
authored
91 }`: consume `item` from the `:example` queue. If the block doesn't
92 complete successfully, the item will be left at a backup queue.
93
0ccf89b @frodsan Update documentation.
frodsan authored
94 `Ost.stop`: halt processing for all queues.
95
182f69f @soveran Update README.
authored
96 `Ost[:example].stop`: halt processing for the `example` queue.
d71040e @soveran Initial import.
authored
97
98 Failures
99 ========
100
182f69f @soveran Update README.
authored
101 **Ost** stores in-process items in backup queues. That allows the
102 developer to deal with exceptions in a way that results adequate
103 for his application.
d71040e @soveran Initial import.
authored
104
182f69f @soveran Update README.
authored
105 There is one backup queue for each worker, with the following
106 convention for naming the key in Redis: given a worker using the
107 `:events` queue, running in the hostname `domU-12-31-39-04-49-C7`
108 with the process id `28431`, the key for the backup queue will be
109 `ost:events:domU-12-31-39-04-49-C7:28431`.
d71040e @soveran Initial import.
authored
110
182f69f @soveran Update README.
authored
111 Here's the explanation for each part:
d71040e @soveran Initial import.
authored
112
182f69f @soveran Update README.
authored
113 * `ost`: namespace for all **Ost** related keys.
114 * `events`: name of the queue.
115 * `domU-12-31-39-04-49-C7`: hostname of the worker.
116 * `28431`: process id of the worker.
d71040e @soveran Initial import.
authored
117
182f69f @soveran Update README.
authored
118 Priorities
119 ----------
d71040e @soveran Initial import.
authored
120
182f69f @soveran Update README.
authored
121 There's no concept of priorities, as each queue is specialized and you
122 can create as many as you want. For example, nothing prevents the
123 creation of the `:example_high_priority` or the
124 `:example_low_priority` queues.
d71040e @soveran Initial import.
authored
125
126 Differences with Delayed::Job and Resque
182f69f @soveran Update README.
authored
127 ----------------------------------------
d71040e @soveran Initial import.
authored
128
182f69f @soveran Update README.
authored
129 Both [Delayed::Job](http://github.com/tobi/delayed_job) and
130 [Resque](http://github.com/defunkt/resque) provide queues and workers
131 (the latter using Redis). They provide dumb workers that process jobs,
132 which are specialized for each task. The specialization takes place
133 in the application side, and the job is serialized and pushed into a
134 queue.
d71040e @soveran Initial import.
authored
135
182f69f @soveran Update README.
authored
136 **Ost**, by contrast, just pushes numbers into specialized queues, and
137 uses workers that are subscribed to specific queues and know what to
138 do with the items they get. The total sum of logic is about the same,
139 but there's less communication and less data transfer with **Ost**.
d71040e @soveran Initial import.
authored
140
141 Installation
142 ------------
143
8fc0714 @soveran Remove sudo from installation instructions.
authored
144 $ gem install ost
d71040e @soveran Initial import.
authored
145
0ccf89b @frodsan Update documentation.
frodsan authored
146 [redic]: https://github.com/amakawa/redic
Something went wrong with that request. Please try again.