Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 475 lines (340 sloc) 18.364 kb
0c1b88b lots more docs.
Robey Pointer authored
1
62482e2 cleanup.
Robey Pointer authored
2 A working guide to kestrel
3 ==========================
0c1b88b lots more docs.
Robey Pointer authored
4
5 Kestrel is a very simple message queue that runs on the JVM and uses the
6 memcache protocol (with some extensions) to talk to clients.
7
8 A single kestrel server has a set of queues identified by a name, which is
9 also the filename of that queue's journal file (usually in
10 `/var/spool/kestrel`). Each queue is a strictly-ordered FIFO of "items" of
11 binary data. Usually this data is in some serialized format like JSON or
12 ruby's marshal format.
13
d6199a9 explain queue names.
Robey Pointer authored
14 Generally queue names should be limited to alphanumerics `[A-Za-z0-9]`, dash
15 (`-`) and underline (`_`). In practice, kestrel doesn't enforce any
16 restrictions other than the name can't contain slash (`/`) because that can't
17 be used in filenames, squiggle (`~`) because it's used for temporary files,
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
18 plus (`+`) because it's used for fanout queues, and dot (`.`) because it's
19 reserved for future use. Queue names are case-sensitive, but if you're running
20 kestrel on OS X or Windows, you will want to refrain from taking advantage of
21 this, since the journal filenames on those two platforms are *not*
22 case-sensitive.
d6199a9 explain queue names.
Robey Pointer authored
23
0c1b88b lots more docs.
Robey Pointer authored
24 A cluster of kestrel servers is like a memcache cluster: the servers don't
25 know about each other, and don't do any cross-communication, so you can add as
26 many as you like. Clients have a list of all servers in the cluster, and pick
27 one at random for each operation. In this way, each queue appears to be spread
8c93d94 more doc improvements.
Robey Pointer authored
28 out across every server, with items in a loose ordering.
0c1b88b lots more docs.
Robey Pointer authored
29
58746d2 more docs!
Robey Pointer authored
30 When kestrel starts up, it scans the journal folder and creates queues based
31 on any journal files it finds there, to restore state to the way it was when
32 it last shutdown (or was killed or died). New queues are created by referring
33 to them (for example, adding or trying to remove an item). A queue can be
34 deleted with the "delete" command.
35
0c1b88b lots more docs.
Robey Pointer authored
36
37 Configuration
38 -------------
39
40 All of the per-queue configuration can be set in the global scope of
8c93d94 more doc improvements.
Robey Pointer authored
41 `production.conf` as a default for all queues, or in the per-queue
0c1b88b lots more docs.
Robey Pointer authored
42 configuration to override the defaults for a specific queue. You can see an
43 example of this in the default config file.
44
45 To confirm the current configuration of each queue, send "dump_config" to
46 a server (which can be done over telnet).
47
48 To reload the config file on a running server, send "reload" the same way.
49 You should immediately see the changes in "dump_config", to confirm.
50
51 - `max_items` (infinite)
52
53 Set a hard limit on the number of items this queue can hold. When the queue
54 is full, `discard_old_when_full` dictates the behavior when a client
55 attempts to add another item.
56
57 - `max_size` (infinite)
58
59 Set a hard limit on the number of bytes (of data in queued items) this
60 queue can hold. When the queue is full, `discard_old_when_full` dictates
61 the behavior when a client attempts to add another item.
62
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
63 - `max_item_size` (infinite)
64
65 Set a hard limit on the number of bytes a single queued item can contain.
66 An add request for an item larger than this will be rejected.
67
0c1b88b lots more docs.
Robey Pointer authored
68 - `discard_old_when_full` (false)
69
70 If this is false, when a queue is full, clients attempting to add another
71 item will get an error. No new items will be accepted. If this is true, old
72 items will be discarded to make room for the new one. This settting has no
73 effect unless at least one of `max_items` or `max_size` is set.
74
75 - `journal` (true)
76
77 If false, don't keep a journal file for this queue. When kestrel exits, any
78 remaining contents in the queue will be lost.
79
80 - `sync_journal` (false)
81
82 If true, sync the journal file on disk after each write. This is usually
83 not necessary but is available for the paranoid. It will probably reduce
84 the maximum throughput of the server.
85
86 - `max_journal_size` (16MB)
87
88 When a journal reaches this size, it will be rolled over to a new file as
89 soon as the queue is empty. The value must be given in bytes.
90
91 - `max_journal_overflow` (10)
92
93 If a journal file grows to this many times its desired maximum size, and
94 the total queue contents (in bytes) are smaller than the desired maximum
95 size, the journal file will be rewritten from scratch, to avoid using up
96 all disk space. For example, using the default `max_journal_size` of 16MB
97 and `max_journal_overflow` of 10, if the journal file ever grows beyond
98 160MB (and the queue's contents are less than 16MB), the journal file will
99 be re-written.
100
101 - `max_memory_size` (128MB)
102
103 If a queue's contents grow past this size, only this part will be kept in
104 memory. Newly added items will be written directly to the journal file and
105 read back into memory as the queue is drained. This setting is a release
106 valve to keep a backed-up queue from consuming all memory. The value must
107 be given in bytes.
108
109 - `max_age` (0 = off)
110
111 Expiration time (in milliseconds) for items on this queue. Any item that
112 has been sitting on the queue longer than this amount will be discarded.
113 Clients may also attach an expiration time when adding items to a queue,
114 but if the expiration time is longer than `max_age`, `max_age` will be
115 used instead.
116
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
117 - `move_expired_to` (none)
118
119 Name of a queue to add expired items to. If set, expired items are added to
120 the requested queue as if by a `SET` command. This can be used to implement
121 special processing for expired items, or to implement a simple "delayed
122 processing" queue.
123
0c1b88b lots more docs.
Robey Pointer authored
124
125 The journal file
126 ----------------
127
128 The journal file is the only on-disk storage of a queue's contents, and it's
129 just a sequential record of each add or remove operation that's happened on
130 that queue. When kestrel starts up, it replays each queue's journal to build
131 up the in-memory queue that it uses for client queries.
132
58746d2 more docs!
Robey Pointer authored
133 The journal file is rotated in one of two conditions:
0c1b88b lots more docs.
Robey Pointer authored
134
135 1. the queue is empty and the journal is larger than `max_journal_size`
136
137 2. the queue is smaller than `max_journal_size` but the journal is larger
138 than `max_journal_overflow` times `max_journal_size`
139
140 For example, if `max_journal_size` is 16MB (the default), and
141 `max_journal_overflow` is 10 (also the default), then if the queue is empty
142 and the journal is larger than 16MB, it will be rotated into a new (empty)
143 file. If the queue is smaller than 16MB, but the journal is larger than 160MB,
144 the journal will be rotated to contain just the live items.
145
146 You can turn the journal off for a queue (`journal` = false) and the queue
147 will exist only in memory. If the server restarts, all enqueued items are
148 lost. You can also force a queue's journal to be sync'd to disk after every
149 write operation (`sync_journal` = true) at a performance cost.
150
151 If a queue grows past `max_memory_size` bytes (128MB by default), only the
152 first 128MB is kept in memory. The journal is used to track later items, and
153 as items are removed, the journal is played forward to keep 128MB in memory.
154 This is usually known as "read-behind" mode, but Twitter engineers sometimes
155 refer to it as the "square snake" because of the diagram used to brainstorm
156 the implementation. When a queue is in read-behind mode, removing an item will
157 often cause 2 disk operations instead of one: one to record the remove, and
158 one to read an item in from disk to keep 128MB in memory. This is the
159 trade-off to avoid filling memory and crashing the JVM.
160
161
58746d2 more docs!
Robey Pointer authored
162 Item expiration
163 ---------------
0c1b88b lots more docs.
Robey Pointer authored
164
58746d2 more docs!
Robey Pointer authored
165 When they come from a client, expiration times are handled in the same way as
166 memcache: if the number is small (less than one million), it's interpreted as
167 a relative number of seconds from now. Otherwise it's interpreted as an
168 absolute unix epoch time, in seconds since the beginning of 1 January 1970
169 GMT.
0c1b88b lots more docs.
Robey Pointer authored
170
58746d2 more docs!
Robey Pointer authored
171 Expiration times are immediately translated into an absolute time, in
172 *milliseconds*, and if it's further in the future than the queue's `max_age`,
173 the `max_age` is used instead. An expiration of 0, which is usually the
174 default, means an item never expires.
0c1b88b lots more docs.
Robey Pointer authored
175
58746d2 more docs!
Robey Pointer authored
176 Expired items are flushed from a queue whenever a new item is added or
177 removed. An idle queue won't have any items expired, but you can trigger a
178 check by doing a "peek" on it.
0c1b88b lots more docs.
Robey Pointer authored
179
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
180 The global config option `expiration_timer_frequency_seconds` can be used to
181 start a background thread that periodically removes expired items from the
182 head of each queue. See the `README.md` file for more.
183
0c1b88b lots more docs.
Robey Pointer authored
184
58746d2 more docs!
Robey Pointer authored
185 Fanout Queues
186 -------------
187
188 If a queue name has a `+` in it (like "`orders+audit`"), it's treated as a
189 fanout queue, using the format `<parent>+<child>`. These queues belong to a
190 parent queue -- in this example, the "orders" queue. Every item written into
191 a parent queue will also be written into each of its children.
192
193 Fanout queues each have their own journal file (if the parent queue has a
194 journal file) and otherwise behave exactly like any other queue. You can get
195 and peek and even add items directly to a child queue if you want. It uses the
196 parent queue's configuration instead of having independent child queue
197 configuration blocks.
198
199 When a fanout queue is first referenced by a client, the journal file (if any)
200 is created, and it will start receiving new items written to the parent queue.
201 Existing items are not copied over. A fanout queue can be deleted to stop it
202 from receiving new items.
203
204
205 Memcache commands
206 -----------------
207
208 - `SET <queue-name> <flags (ignored)> <expiration> <# bytes>`
209
210 Add an item to a queue. It may fail if the queue has a size or item limit
211 and it's full.
212
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
213 - `GET <queue-name>[options]`
66c498b docs!
Robey Pointer authored
214
58746d2 more docs!
Robey Pointer authored
215 Remove an item from a queue. It will return an empty response immediately if
216 the queue is empty. The queue name may be followed by options separated
217 by `/`:
218
66c498b docs!
Robey Pointer authored
219 - `/t=<milliseconds>`
220
221 Wait up to a given time limit for a new item to arrive. If an item arrives
222 on the queue within this timeout, it's returned as normal. Otherwise,
223 after that timeout, an empty response is returned.
224
225 - `/open`
226
227 Tentatively remove an item from the queue. The item is returned as usual
228 but is also set aside in case the client disappears before sending a
229 "close" request. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
230
66c498b docs!
Robey Pointer authored
231 - `/close`
58746d2 more docs!
Robey Pointer authored
232
66c498b docs!
Robey Pointer authored
233 Close any existing open read. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
234
66c498b docs!
Robey Pointer authored
235 - `/abort`
58746d2 more docs!
Robey Pointer authored
236
66c498b docs!
Robey Pointer authored
237 Cancel any existing open read, returing that item to the head of the
238 queue. It will be the next item fetched. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
239
66c498b docs!
Robey Pointer authored
240 - `/peek`
58746d2 more docs!
Robey Pointer authored
241
66c498b docs!
Robey Pointer authored
242 Return the first available item from the queue, if there is one, but don't
243 remove it. You can't combine this with any of the reliable read options.
58746d2 more docs!
Robey Pointer authored
244
d1b4ee7 R. Tyler Croy Update guide.md
rtyler authored
245 For example, to open a new read, waiting up to 500msec for an item:
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
246
247 GET work/t=500/open
248
249 Or to close an existing read and open a new one:
250
251 GET work/close/open
252
58746d2 more docs!
Robey Pointer authored
253 - `DELETE <queue-name>`
254
255 Drop a queue, discarding any items in it, and deleting any associated
256 journal files.
257
258 - `FLUSH <queue-name>`
259
260 Discard all items remaining in this queue. The queue remains live and new
261 items can be added. The time it takes to flush will be linear to the current
262 queue size, and any other activity on this queue will block while it's being
263 flushed.
264
265 - `FLUSH_ALL`
266
267 Discard all items remaining in all queues. The queues are flushed one at a
268 time, as if kestrel received a `FLUSH` command for each queue.
0c1b88b lots more docs.
Robey Pointer authored
269
58746d2 more docs!
Robey Pointer authored
270 - `VERSION`
271
272 Display the kestrel version in a way compatible with memcache.
273
274 - `SHUTDOWN`
275
276 Cleanly shutdown the server and exit.
277
278 - `RELOAD`
279
280 Reload the config file and reconfigure all queues. This should have no
281 noticable effect on the server's responsiveness.
282
283 - `DUMP_CONFIG`
284
285 Dump a list of each queue currently known to the server, and list the config
286 values for each queue. The format is:
287
66c498b docs!
Robey Pointer authored
288 queue 'master' {
289 max_items=2147483647
290 max_size=9223372036854775807
291 max_age=0
292 max_journal_size=16277216
293 max_memory_size=134217728
294 max_journal_overflow=10
295 max_journal_size_absolute=9223372036854775807
296 discard_old_when_full=false
297 journal=true
298 sync_journal=false
299 }
300
301 The last queue will be followed by `END` on a line by itself.
58746d2 more docs!
Robey Pointer authored
302
303 - `STATS`
304
305 Display server stats in memcache style. They're described below.
306
307 - `DUMP_STATS`
308
309 Display server stats in a more readable style, grouped by queue. They're
310 described below.
311
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
312
58746d2 more docs!
Robey Pointer authored
313 Reliable reads
314 --------------
315
316 Normally when a client removes an item from the queue, kestrel immediately
317 discards the item and assumes the client has taken ownership. This isn't
318 always safe, because a client could crash or lose the network connection
319 before it gets the item. So kestrel also supports a "reliable read" that
320 happens in two stages, using the `/open` and `/close` options to `GET`.
321
322 When `/open` is used, and an item is available, kestrel will remove it from
323 the queue and send it to the client as usual. But it will also set the item
324 aside. If a client disconnects while it has an open read, the item is put back
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
325 into the queue, at the head, so it will be the next item fetched. Only one
326 item can be "open" per client connection.
58746d2 more docs!
Robey Pointer authored
327
328 A previous open request is closed with `/close`. The server will reject any
329 attempt to open another read when one is already open, but it will ignore
330 `/close` if there's no open request, so that you can add `/close` to every
331 `GET` request for convenience.
332
333 If for some reason you want to abort a read without disconnecting, you can use
334 `/abort`. But because aborted items are placed back at the head of the queue,
335 this isn't a good way to deal with client errors. Since the error-causing item
336 will always be the next one available, you'll end up bouncing the same item
337 around between clients instead of making progress.
338
339 There's always a trade-off: either potentially lose items or potentially
340 receive the same item multiple times. Reliable reads choose the latter option.
341 To use this tactic successfully, work items should be idempotent, meaning the
342 work could be done 2 or 3 times and have the same effect as if it had been
343 done only once (except wasting some resources).
344
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
345 Example:
346
347 GET dirty_jobs/close/open
348 (receives job 1)
349 GET dirty_jobs/close/open
350 (closes job 1, receives job 2)
351 ...etc...
58746d2 more docs!
Robey Pointer authored
352
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
353
58746d2 more docs!
Robey Pointer authored
354 Server stats
355 ------------
356
66c498b docs!
Robey Pointer authored
357 Global stats reported by kestrel are:
358
359 - `uptime` - seconds the server has been online
360 - `time` - current time in unix epoch
361 - `version` - version string, like "1.2"
362 - `curr_items` - total of items waiting in all queues
363 - `total_itmes` - total of items that have ever been added in this server's
364 lifetime
365 - `bytes` - total byte size of items waiting in all queues
366 - `curr_connections` - current open connections from clients
367 - `total_connections` - total connections that have been opened in this
368 server's lifetime
369 - `cmd_get` - total `GET` requests
370 - `cmd_set` - total `SET` requests
371 - `cmd_peek` - total `GET/peek` requests
372 - `get_hits` - total `GET` requests that received an item
373 - `get_misses` - total `GET` requests on an empty queue
374 - `bytes_read` - total bytes read from clients
375 - `bytes_written` - total bytes written to clients
376
377 For each queue, the following stats are also reported:
378
379 - `items` - items waiting in this queue
380 - `bytes` - total byte size of items waiting in this queue
381 - `total_items` - total items that have been added to this queue in this
382 server's lifetime
383 - `logsize` - byte size of the queue's journal file
384 - `expired_items` - total items that have been expired from this queue in this
385 server's lifetime
386 - `mem_items` - items in this queue that are currently in memory
387 - `mem_bytes` - total byte size of items in this queue that are currently in
388 memory (will always be less than or equal to `max_memory_size` config for
389 the queue)
390 - `age` - time, in milliseconds, that the last item to be fetched from this
391 queue had been waiting; that is, the time between `SET` and `GET`; if the
392 queue is empty, this will always be zero
393 - `discarded` - number of items discarded because the queue was too full
394 - `waiters` - number of clients waiting for an item from this queue (using
395 `GET/t`)
721bf6b document the new stat.
Robey Pointer authored
396 - `open_transactions` - items read with `/open` but not yet confirmed
66c498b docs!
Robey Pointer authored
397
398
399 Kestrel as a library
400 --------------------
401
402 You can use kestrel as a library by just sticking the jar on your classpath.
403 It's a cheap way to get a durable work queue for inter-process or inter-thread
404 communication. Each queue is represented by a `PersistentQueue` object:
405
406 class PersistentQueue(persistencePath: String, val name: String, val config: ConfigMap)
407
408 and must be initialized before using:
409
410 def setup(): Unit
58746d2 more docs!
Robey Pointer authored
411
66c498b docs!
Robey Pointer authored
412 specifying the path for the journal files (if the queue will be journaled),
413 the name of the queue, and a configgy `ConfigMap` block with any special
414 configuration. (See "Configuration" above.)
58746d2 more docs!
Robey Pointer authored
415
66c498b docs!
Robey Pointer authored
416 To add an item to a queue:
58746d2 more docs!
Robey Pointer authored
417
66c498b docs!
Robey Pointer authored
418 def add(value: Array[Byte], expiry: Long): Boolean
58746d2 more docs!
Robey Pointer authored
419
66c498b docs!
Robey Pointer authored
420 It will return `false` if the item was rejected because the queue was full.
0c1b88b lots more docs.
Robey Pointer authored
421
66c498b docs!
Robey Pointer authored
422 Queue items are represented by a case class:
423
424 case class QItem(addTime: Long, expiry: Long, data: Array[Byte], var xid: Int)
425
426 and several operations exist to remove or peek at the head item:
427
428 def peek(): Option[QItem]
429 def remove(): Option[QItem]
430
431 To open a reliable read, set `transaction` true, and later confirm or unremove
432 the item by its `xid`:
433
434 def remove(transaction: Boolean): Option[QItem]
435 def unremove(xid: Int)
436 def confirmRemove(xid: Int)
437
438 You can also asynchronously remove or peek at items using actors, as either
439 a `receive` or `react` callback:
440
441 def removeReact(timeoutAbsolute: Long, transaction: Boolean)(f: Option[QItem] => Unit): Unit
442 def removeReceive(timeoutAbsolute: Long, transaction: Boolean): Option[QItem]
443 def peekReact(timeoutAbsolute: Long)(f: Option[QItem] => Unit): Unit
444 def peekReceive(timeoutAbsolute: Long): Option[QItem]
445
446 When done, you should close the queue:
447
448 def close(): Unit
449 def isClosed: Boolean
450
451 Here's a short example:
452
453 var queue = new PersistentQueue("/var/spool/kestrel", "work", config)
454 queue.setup()
455
456 // add an item with no expiration:
457 queue.add("hello".getBytes, 0)
458
459 // start to remove it, then back out:
460 val item = queue.remove(true)
461 queue.unremove(item.xid)
462
463 // remove an item with a 500msec timeout, and confirm it:
464 queue.removeReact(System.currentTimeMillis + 500, true) { x =>
465 x match {
466 case None =>
467 println("nothing. :(")
468 case Some(item) =>
469 println("got: " + new String(item.data))
470 queue.confirmRemove(item.xid)
471 }
472 }
0c1b88b lots more docs.
Robey Pointer authored
473
66c498b docs!
Robey Pointer authored
474 queue.close()
Something went wrong with that request. Please try again.