Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 457 lines (329 sloc) 17.702 kB
0c1b88b lots more docs.
Robey Pointer authored
1
62482e2 cleanup.
Robey Pointer authored
2 A working guide to kestrel
3 ==========================
0c1b88b lots more docs.
Robey Pointer authored
4
5 Kestrel is a very simple message queue that runs on the JVM and uses the
6 memcache protocol (with some extensions) to talk to clients.
7
8 A single kestrel server has a set of queues identified by a name, which is
9 also the filename of that queue's journal file (usually in
10 `/var/spool/kestrel`). Each queue is a strictly-ordered FIFO of "items" of
11 binary data. Usually this data is in some serialized format like JSON or
12 ruby's marshal format.
13
d6199a9 explain queue names.
Robey Pointer authored
14 Generally queue names should be limited to alphanumerics `[A-Za-z0-9]`, dash
15 (`-`) and underline (`_`). In practice, kestrel doesn't enforce any
16 restrictions other than the name can't contain slash (`/`) because that can't
17 be used in filenames, squiggle (`~`) because it's used for temporary files,
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
18 plus (`+`) because it's used for fanout queues, and dot (`.`) because it's
19 reserved for future use. Queue names are case-sensitive, but if you're running
20 kestrel on OS X or Windows, you will want to refrain from taking advantage of
21 this, since the journal filenames on those two platforms are *not*
22 case-sensitive.
d6199a9 explain queue names.
Robey Pointer authored
23
0c1b88b lots more docs.
Robey Pointer authored
24 A cluster of kestrel servers is like a memcache cluster: the servers don't
25 know about each other, and don't do any cross-communication, so you can add as
26 many as you like. Clients have a list of all servers in the cluster, and pick
27 one at random for each operation. In this way, each queue appears to be spread
8c93d94 more doc improvements.
Robey Pointer authored
28 out across every server, with items in a loose ordering.
0c1b88b lots more docs.
Robey Pointer authored
29
58746d2 more docs!
Robey Pointer authored
30 When kestrel starts up, it scans the journal folder and creates queues based
31 on any journal files it finds there, to restore state to the way it was when
32 it last shutdown (or was killed or died). New queues are created by referring
33 to them (for example, adding or trying to remove an item). A queue can be
34 deleted with the "delete" command.
35
0c1b88b lots more docs.
Robey Pointer authored
36
37 Configuration
38 -------------
39
40 All of the per-queue configuration can be set in the global scope of
8c93d94 more doc improvements.
Robey Pointer authored
41 `production.conf` as a default for all queues, or in the per-queue
0c1b88b lots more docs.
Robey Pointer authored
42 configuration to override the defaults for a specific queue. You can see an
43 example of this in the default config file.
44
45 To confirm the current configuration of each queue, send "dump_config" to
46 a server (which can be done over telnet).
47
48 To reload the config file on a running server, send "reload" the same way.
49 You should immediately see the changes in "dump_config", to confirm.
50
51 - `max_items` (infinite)
52
53 Set a hard limit on the number of items this queue can hold. When the queue
54 is full, `discard_old_when_full` dictates the behavior when a client
55 attempts to add another item.
56
57 - `max_size` (infinite)
58
59 Set a hard limit on the number of bytes (of data in queued items) this
60 queue can hold. When the queue is full, `discard_old_when_full` dictates
61 the behavior when a client attempts to add another item.
62
63 - `discard_old_when_full` (false)
64
65 If this is false, when a queue is full, clients attempting to add another
66 item will get an error. No new items will be accepted. If this is true, old
67 items will be discarded to make room for the new one. This settting has no
68 effect unless at least one of `max_items` or `max_size` is set.
69
70 - `journal` (true)
71
72 If false, don't keep a journal file for this queue. When kestrel exits, any
73 remaining contents in the queue will be lost.
74
75 - `sync_journal` (false)
76
77 If true, sync the journal file on disk after each write. This is usually
78 not necessary but is available for the paranoid. It will probably reduce
79 the maximum throughput of the server.
80
81 - `max_journal_size` (16MB)
82
83 When a journal reaches this size, it will be rolled over to a new file as
84 soon as the queue is empty. The value must be given in bytes.
85
86 - `max_journal_overflow` (10)
87
88 If a journal file grows to this many times its desired maximum size, and
89 the total queue contents (in bytes) are smaller than the desired maximum
90 size, the journal file will be rewritten from scratch, to avoid using up
91 all disk space. For example, using the default `max_journal_size` of 16MB
92 and `max_journal_overflow` of 10, if the journal file ever grows beyond
93 160MB (and the queue's contents are less than 16MB), the journal file will
94 be re-written.
95
96 - `max_memory_size` (128MB)
97
98 If a queue's contents grow past this size, only this part will be kept in
99 memory. Newly added items will be written directly to the journal file and
100 read back into memory as the queue is drained. This setting is a release
101 valve to keep a backed-up queue from consuming all memory. The value must
102 be given in bytes.
103
104 - `max_age` (0 = off)
105
106 Expiration time (in milliseconds) for items on this queue. Any item that
107 has been sitting on the queue longer than this amount will be discarded.
108 Clients may also attach an expiration time when adding items to a queue,
109 but if the expiration time is longer than `max_age`, `max_age` will be
110 used instead.
111
112
113 The journal file
114 ----------------
115
116 The journal file is the only on-disk storage of a queue's contents, and it's
117 just a sequential record of each add or remove operation that's happened on
118 that queue. When kestrel starts up, it replays each queue's journal to build
119 up the in-memory queue that it uses for client queries.
120
58746d2 more docs!
Robey Pointer authored
121 The journal file is rotated in one of two conditions:
0c1b88b lots more docs.
Robey Pointer authored
122
123 1. the queue is empty and the journal is larger than `max_journal_size`
124
125 2. the queue is smaller than `max_journal_size` but the journal is larger
126 than `max_journal_overflow` times `max_journal_size`
127
128 For example, if `max_journal_size` is 16MB (the default), and
129 `max_journal_overflow` is 10 (also the default), then if the queue is empty
130 and the journal is larger than 16MB, it will be rotated into a new (empty)
131 file. If the queue is smaller than 16MB, but the journal is larger than 160MB,
132 the journal will be rotated to contain just the live items.
133
134 You can turn the journal off for a queue (`journal` = false) and the queue
135 will exist only in memory. If the server restarts, all enqueued items are
136 lost. You can also force a queue's journal to be sync'd to disk after every
137 write operation (`sync_journal` = true) at a performance cost.
138
139 If a queue grows past `max_memory_size` bytes (128MB by default), only the
140 first 128MB is kept in memory. The journal is used to track later items, and
141 as items are removed, the journal is played forward to keep 128MB in memory.
142 This is usually known as "read-behind" mode, but Twitter engineers sometimes
143 refer to it as the "square snake" because of the diagram used to brainstorm
144 the implementation. When a queue is in read-behind mode, removing an item will
145 often cause 2 disk operations instead of one: one to record the remove, and
146 one to read an item in from disk to keep 128MB in memory. This is the
147 trade-off to avoid filling memory and crashing the JVM.
148
149
58746d2 more docs!
Robey Pointer authored
150 Item expiration
151 ---------------
0c1b88b lots more docs.
Robey Pointer authored
152
58746d2 more docs!
Robey Pointer authored
153 When they come from a client, expiration times are handled in the same way as
154 memcache: if the number is small (less than one million), it's interpreted as
155 a relative number of seconds from now. Otherwise it's interpreted as an
156 absolute unix epoch time, in seconds since the beginning of 1 January 1970
157 GMT.
0c1b88b lots more docs.
Robey Pointer authored
158
58746d2 more docs!
Robey Pointer authored
159 Expiration times are immediately translated into an absolute time, in
160 *milliseconds*, and if it's further in the future than the queue's `max_age`,
161 the `max_age` is used instead. An expiration of 0, which is usually the
162 default, means an item never expires.
0c1b88b lots more docs.
Robey Pointer authored
163
58746d2 more docs!
Robey Pointer authored
164 Expired items are flushed from a queue whenever a new item is added or
165 removed. An idle queue won't have any items expired, but you can trigger a
166 check by doing a "peek" on it.
0c1b88b lots more docs.
Robey Pointer authored
167
168
58746d2 more docs!
Robey Pointer authored
169 Fanout Queues
170 -------------
171
172 If a queue name has a `+` in it (like "`orders+audit`"), it's treated as a
173 fanout queue, using the format `<parent>+<child>`. These queues belong to a
174 parent queue -- in this example, the "orders" queue. Every item written into
175 a parent queue will also be written into each of its children.
176
177 Fanout queues each have their own journal file (if the parent queue has a
178 journal file) and otherwise behave exactly like any other queue. You can get
179 and peek and even add items directly to a child queue if you want. It uses the
180 parent queue's configuration instead of having independent child queue
181 configuration blocks.
182
183 When a fanout queue is first referenced by a client, the journal file (if any)
184 is created, and it will start receiving new items written to the parent queue.
185 Existing items are not copied over. A fanout queue can be deleted to stop it
186 from receiving new items.
187
188
189 Memcache commands
190 -----------------
191
192 - `SET <queue-name> <flags (ignored)> <expiration> <# bytes>`
193
194 Add an item to a queue. It may fail if the queue has a size or item limit
195 and it's full.
196
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
197 - `GET <queue-name>[options]`
66c498b docs!
Robey Pointer authored
198
58746d2 more docs!
Robey Pointer authored
199 Remove an item from a queue. It will return an empty response immediately if
200 the queue is empty. The queue name may be followed by options separated
201 by `/`:
202
66c498b docs!
Robey Pointer authored
203 - `/t=<milliseconds>`
204
205 Wait up to a given time limit for a new item to arrive. If an item arrives
206 on the queue within this timeout, it's returned as normal. Otherwise,
207 after that timeout, an empty response is returned.
208
209 - `/open`
210
211 Tentatively remove an item from the queue. The item is returned as usual
212 but is also set aside in case the client disappears before sending a
213 "close" request. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
214
66c498b docs!
Robey Pointer authored
215 - `/close`
58746d2 more docs!
Robey Pointer authored
216
66c498b docs!
Robey Pointer authored
217 Close any existing open read. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
218
66c498b docs!
Robey Pointer authored
219 - `/abort`
58746d2 more docs!
Robey Pointer authored
220
66c498b docs!
Robey Pointer authored
221 Cancel any existing open read, returing that item to the head of the
222 queue. It will be the next item fetched. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
223
66c498b docs!
Robey Pointer authored
224 - `/peek`
58746d2 more docs!
Robey Pointer authored
225
66c498b docs!
Robey Pointer authored
226 Return the first available item from the queue, if there is one, but don't
227 remove it. You can't combine this with any of the reliable read options.
58746d2 more docs!
Robey Pointer authored
228
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
229 For example, to open a new read, waiting up to 250msec for an item:
230
231 GET work/t=500/open
232
233 Or to close an existing read and open a new one:
234
235 GET work/close/open
236
58746d2 more docs!
Robey Pointer authored
237 - `DELETE <queue-name>`
238
239 Drop a queue, discarding any items in it, and deleting any associated
240 journal files.
241
242 - `FLUSH <queue-name>`
243
244 Discard all items remaining in this queue. The queue remains live and new
245 items can be added. The time it takes to flush will be linear to the current
246 queue size, and any other activity on this queue will block while it's being
247 flushed.
248
249 - `FLUSH_ALL`
250
251 Discard all items remaining in all queues. The queues are flushed one at a
252 time, as if kestrel received a `FLUSH` command for each queue.
0c1b88b lots more docs.
Robey Pointer authored
253
58746d2 more docs!
Robey Pointer authored
254 - `VERSION`
255
256 Display the kestrel version in a way compatible with memcache.
257
258 - `SHUTDOWN`
259
260 Cleanly shutdown the server and exit.
261
262 - `RELOAD`
263
264 Reload the config file and reconfigure all queues. This should have no
265 noticable effect on the server's responsiveness.
266
267 - `DUMP_CONFIG`
268
269 Dump a list of each queue currently known to the server, and list the config
270 values for each queue. The format is:
271
66c498b docs!
Robey Pointer authored
272 queue 'master' {
273 max_items=2147483647
274 max_size=9223372036854775807
275 max_age=0
276 max_journal_size=16277216
277 max_memory_size=134217728
278 max_journal_overflow=10
279 max_journal_size_absolute=9223372036854775807
280 discard_old_when_full=false
281 journal=true
282 sync_journal=false
283 }
284
285 The last queue will be followed by `END` on a line by itself.
58746d2 more docs!
Robey Pointer authored
286
287 - `STATS`
288
289 Display server stats in memcache style. They're described below.
290
291 - `DUMP_STATS`
292
293 Display server stats in a more readable style, grouped by queue. They're
294 described below.
295
296 Reliable reads
297 --------------
298
299 Normally when a client removes an item from the queue, kestrel immediately
300 discards the item and assumes the client has taken ownership. This isn't
301 always safe, because a client could crash or lose the network connection
302 before it gets the item. So kestrel also supports a "reliable read" that
303 happens in two stages, using the `/open` and `/close` options to `GET`.
304
305 When `/open` is used, and an item is available, kestrel will remove it from
306 the queue and send it to the client as usual. But it will also set the item
307 aside. If a client disconnects while it has an open read, the item is put back
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
308 into the queue, at the head, so it will be the next item fetched. Only one
309 item can be "open" per client connection.
58746d2 more docs!
Robey Pointer authored
310
311 A previous open request is closed with `/close`. The server will reject any
312 attempt to open another read when one is already open, but it will ignore
313 `/close` if there's no open request, so that you can add `/close` to every
314 `GET` request for convenience.
315
316 If for some reason you want to abort a read without disconnecting, you can use
317 `/abort`. But because aborted items are placed back at the head of the queue,
318 this isn't a good way to deal with client errors. Since the error-causing item
319 will always be the next one available, you'll end up bouncing the same item
320 around between clients instead of making progress.
321
322 There's always a trade-off: either potentially lose items or potentially
323 receive the same item multiple times. Reliable reads choose the latter option.
324 To use this tactic successfully, work items should be idempotent, meaning the
325 work could be done 2 or 3 times and have the same effect as if it had been
326 done only once (except wasting some resources).
327
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
328 Example:
329
330 GET dirty_jobs/close/open
331 (receives job 1)
332 GET dirty_jobs/close/open
333 (closes job 1, receives job 2)
334 ...etc...
58746d2 more docs!
Robey Pointer authored
335
336 Server stats
337 ------------
338
66c498b docs!
Robey Pointer authored
339 Global stats reported by kestrel are:
340
341 - `uptime` - seconds the server has been online
342 - `time` - current time in unix epoch
343 - `version` - version string, like "1.2"
344 - `curr_items` - total of items waiting in all queues
345 - `total_itmes` - total of items that have ever been added in this server's
346 lifetime
347 - `bytes` - total byte size of items waiting in all queues
348 - `curr_connections` - current open connections from clients
349 - `total_connections` - total connections that have been opened in this
350 server's lifetime
351 - `cmd_get` - total `GET` requests
352 - `cmd_set` - total `SET` requests
353 - `cmd_peek` - total `GET/peek` requests
354 - `get_hits` - total `GET` requests that received an item
355 - `get_misses` - total `GET` requests on an empty queue
356 - `bytes_read` - total bytes read from clients
357 - `bytes_written` - total bytes written to clients
358
359 For each queue, the following stats are also reported:
360
361 - `items` - items waiting in this queue
362 - `bytes` - total byte size of items waiting in this queue
363 - `total_items` - total items that have been added to this queue in this
364 server's lifetime
365 - `logsize` - byte size of the queue's journal file
366 - `expired_items` - total items that have been expired from this queue in this
367 server's lifetime
368 - `mem_items` - items in this queue that are currently in memory
369 - `mem_bytes` - total byte size of items in this queue that are currently in
370 memory (will always be less than or equal to `max_memory_size` config for
371 the queue)
372 - `age` - time, in milliseconds, that the last item to be fetched from this
373 queue had been waiting; that is, the time between `SET` and `GET`; if the
374 queue is empty, this will always be zero
375 - `discarded` - number of items discarded because the queue was too full
376 - `waiters` - number of clients waiting for an item from this queue (using
377 `GET/t`)
721bf6b document the new stat.
Robey Pointer authored
378 - `open_transactions` - items read with `/open` but not yet confirmed
66c498b docs!
Robey Pointer authored
379
380
381 Kestrel as a library
382 --------------------
383
384 You can use kestrel as a library by just sticking the jar on your classpath.
385 It's a cheap way to get a durable work queue for inter-process or inter-thread
386 communication. Each queue is represented by a `PersistentQueue` object:
387
388 class PersistentQueue(persistencePath: String, val name: String, val config: ConfigMap)
389
390 and must be initialized before using:
391
392 def setup(): Unit
58746d2 more docs!
Robey Pointer authored
393
66c498b docs!
Robey Pointer authored
394 specifying the path for the journal files (if the queue will be journaled),
395 the name of the queue, and a configgy `ConfigMap` block with any special
396 configuration. (See "Configuration" above.)
58746d2 more docs!
Robey Pointer authored
397
66c498b docs!
Robey Pointer authored
398 To add an item to a queue:
58746d2 more docs!
Robey Pointer authored
399
66c498b docs!
Robey Pointer authored
400 def add(value: Array[Byte], expiry: Long): Boolean
58746d2 more docs!
Robey Pointer authored
401
66c498b docs!
Robey Pointer authored
402 It will return `false` if the item was rejected because the queue was full.
0c1b88b lots more docs.
Robey Pointer authored
403
66c498b docs!
Robey Pointer authored
404 Queue items are represented by a case class:
405
406 case class QItem(addTime: Long, expiry: Long, data: Array[Byte], var xid: Int)
407
408 and several operations exist to remove or peek at the head item:
409
410 def peek(): Option[QItem]
411 def remove(): Option[QItem]
412
413 To open a reliable read, set `transaction` true, and later confirm or unremove
414 the item by its `xid`:
415
416 def remove(transaction: Boolean): Option[QItem]
417 def unremove(xid: Int)
418 def confirmRemove(xid: Int)
419
420 You can also asynchronously remove or peek at items using actors, as either
421 a `receive` or `react` callback:
422
423 def removeReact(timeoutAbsolute: Long, transaction: Boolean)(f: Option[QItem] => Unit): Unit
424 def removeReceive(timeoutAbsolute: Long, transaction: Boolean): Option[QItem]
425 def peekReact(timeoutAbsolute: Long)(f: Option[QItem] => Unit): Unit
426 def peekReceive(timeoutAbsolute: Long): Option[QItem]
427
428 When done, you should close the queue:
429
430 def close(): Unit
431 def isClosed: Boolean
432
433 Here's a short example:
434
435 var queue = new PersistentQueue("/var/spool/kestrel", "work", config)
436 queue.setup()
437
438 // add an item with no expiration:
439 queue.add("hello".getBytes, 0)
440
441 // start to remove it, then back out:
442 val item = queue.remove(true)
443 queue.unremove(item.xid)
444
445 // remove an item with a 500msec timeout, and confirm it:
446 queue.removeReact(System.currentTimeMillis + 500, true) { x =>
447 x match {
448 case None =>
449 println("nothing. :(")
450 case Some(item) =>
451 println("got: " + new String(item.data))
452 queue.confirmRemove(item.xid)
453 }
454 }
0c1b88b lots more docs.
Robey Pointer authored
455
66c498b docs!
Robey Pointer authored
456 queue.close()
Something went wrong with that request. Please try again.