Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 468 lines (340 sloc) 18.552 kb
0c1b88b lots more docs.
Robey Pointer authored
1
62482e2 cleanup.
Robey Pointer authored
2 A working guide to kestrel
3 ==========================
0c1b88b lots more docs.
Robey Pointer authored
4
5 Kestrel is a very simple message queue that runs on the JVM and uses the
6 memcache protocol (with some extensions) to talk to clients.
7
8 A single kestrel server has a set of queues identified by a name, which is
9 also the filename of that queue's journal file (usually in
10 `/var/spool/kestrel`). Each queue is a strictly-ordered FIFO of "items" of
11 binary data. Usually this data is in some serialized format like JSON or
12 ruby's marshal format.
13
d6199a9 explain queue names.
Robey Pointer authored
14 Generally queue names should be limited to alphanumerics `[A-Za-z0-9]`, dash
15 (`-`) and underline (`_`). In practice, kestrel doesn't enforce any
16 restrictions other than the name can't contain slash (`/`) because that can't
17 be used in filenames, squiggle (`~`) because it's used for temporary files,
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
18 plus (`+`) because it's used for fanout queues, and dot (`.`) because it's
19 reserved for future use. Queue names are case-sensitive, but if you're running
20 kestrel on OS X or Windows, you will want to refrain from taking advantage of
21 this, since the journal filenames on those two platforms are *not*
22 case-sensitive.
d6199a9 explain queue names.
Robey Pointer authored
23
0c1b88b lots more docs.
Robey Pointer authored
24 A cluster of kestrel servers is like a memcache cluster: the servers don't
25 know about each other, and don't do any cross-communication, so you can add as
26 many as you like. Clients have a list of all servers in the cluster, and pick
27 one at random for each operation. In this way, each queue appears to be spread
8c93d94 more doc improvements.
Robey Pointer authored
28 out across every server, with items in a loose ordering.
0c1b88b lots more docs.
Robey Pointer authored
29
58746d2 more docs!
Robey Pointer authored
30 When kestrel starts up, it scans the journal folder and creates queues based
31 on any journal files it finds there, to restore state to the way it was when
32 it last shutdown (or was killed or died). New queues are created by referring
33 to them (for example, adding or trying to remove an item). A queue can be
34 deleted with the "delete" command.
35
0c1b88b lots more docs.
Robey Pointer authored
36
37 Configuration
38 -------------
39
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
40 The config files for kestrel are scala expressions loaded at runtime, usually
41 from `production.scala`, although you can use `development.scala` by passing
42 `-Dstage=development` to the java command line.
43
44 The config file evaluates to a `KestrelConfig` object that's used to configure
45 the server as a whole, a default queue, and any overrides for specific named
46 queues. The fields on `KestrelConfig` are documented here with their default
47 values:
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
48 [KestrelConfig.html](http://robey.github.com/kestrel/doc/main/api/net/lag/kestrel/config/KestrelConfig.html)
0c1b88b lots more docs.
Robey Pointer authored
49
50 To confirm the current configuration of each queue, send "dump_config" to
51 a server (which can be done over telnet).
52
53 To reload the config file on a running server, send "reload" the same way.
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
54 You should immediately see the changes in "dump_config", to confirm. Reloading
55 will only affect queue configuration, not global server configuration. To
56 change the server configuration, restart the server.
0c1b88b lots more docs.
Robey Pointer authored
57
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
58 Logging is configured according to `util-logging`. The logging configuration
59 syntax is described here:
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
60 [util-logging](https://github.com/twitter/util/blob/master/util-logging/README.markdown)
0c1b88b lots more docs.
Robey Pointer authored
61
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
62 Per-queue configuration is documented here:
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
63 [QueueBuilder.html](http://robey.github.com/kestrel/doc/main/api/net/lag/kestrel/config/QueueBuilder.html)
64
65
66 Full queues
67 -----------
68
69 A queue can have the following limits set on it:
70
71 - `maxItems` - total items in the queue
72 - `maxSize` - total bytes of data in the items in the queue
73
74 If either of these limits is reached, no new items can be added to the queue.
75 (Clients will receive an error when trying to add.) If you set
76 `discardOldWhenFull` to true, then all adds will succeed, and the oldest
77 item(s) will be silently discarded until the queue is back within the item
78 and size limits.
79
80 `maxItemSize` limits the size of any individual item. If an add is attempted
81 with an item larger than this limit, it always fails.
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
82
0c1b88b lots more docs.
Robey Pointer authored
83
84 The journal file
85 ----------------
86
87 The journal file is the only on-disk storage of a queue's contents, and it's
88 just a sequential record of each add or remove operation that's happened on
89 that queue. When kestrel starts up, it replays each queue's journal to build
90 up the in-memory queue that it uses for client queries.
91
58746d2 more docs!
Robey Pointer authored
92 The journal file is rotated in one of two conditions:
0c1b88b lots more docs.
Robey Pointer authored
93
1531d0d finish cleaning up the guide.
Robey Pointer authored
94 1. the queue is empty and the journal is larger than `defaultJournalSize`
0c1b88b lots more docs.
Robey Pointer authored
95
1531d0d finish cleaning up the guide.
Robey Pointer authored
96 2. the journal is larger than `maxJournalSize`
0c1b88b lots more docs.
Robey Pointer authored
97
1531d0d finish cleaning up the guide.
Robey Pointer authored
98 For example, if `defaultJournalSize` is 16MB (the default), then if the queue
99 is empty and the journal is larger than 16MB, it will be truncated into a new
100 (empty) file. If the journal is larger than `maxJournalSize` (1GB by default),
101 the journal will be rewritten periodically to contain just the live items.
0c1b88b lots more docs.
Robey Pointer authored
102
1531d0d finish cleaning up the guide.
Robey Pointer authored
103 You can turn the journal off for a queue (`keepJournal` = false) and the queue
0c1b88b lots more docs.
Robey Pointer authored
104 will exist only in memory. If the server restarts, all enqueued items are
1531d0d finish cleaning up the guide.
Robey Pointer authored
105 lost. You can also force a queue's journal to be sync'd to disk periodically,
106 or even after every write operation, at a performance cost, using
107 `syncJournal`.
0c1b88b lots more docs.
Robey Pointer authored
108
1531d0d finish cleaning up the guide.
Robey Pointer authored
109 If a queue grows past `maxMemorySize` bytes (128MB by default), only the
0c1b88b lots more docs.
Robey Pointer authored
110 first 128MB is kept in memory. The journal is used to track later items, and
111 as items are removed, the journal is played forward to keep 128MB in memory.
112 This is usually known as "read-behind" mode, but Twitter engineers sometimes
113 refer to it as the "square snake" because of the diagram used to brainstorm
114 the implementation. When a queue is in read-behind mode, removing an item will
115 often cause 2 disk operations instead of one: one to record the remove, and
116 one to read an item in from disk to keep 128MB in memory. This is the
117 trade-off to avoid filling memory and crashing the JVM.
118
119
58746d2 more docs!
Robey Pointer authored
120 Item expiration
121 ---------------
0c1b88b lots more docs.
Robey Pointer authored
122
58746d2 more docs!
Robey Pointer authored
123 When they come from a client, expiration times are handled in the same way as
124 memcache: if the number is small (less than one million), it's interpreted as
125 a relative number of seconds from now. Otherwise it's interpreted as an
126 absolute unix epoch time, in seconds since the beginning of 1 January 1970
127 GMT.
0c1b88b lots more docs.
Robey Pointer authored
128
58746d2 more docs!
Robey Pointer authored
129 Expiration times are immediately translated into an absolute time, in
1531d0d finish cleaning up the guide.
Robey Pointer authored
130 *milliseconds*, and if it's further in the future than the queue's `maxAge`,
a3c112f fix typos
Robey Pointer authored
131 the `maxAge` is used instead. An expiration of 0, which is usually the
58746d2 more docs!
Robey Pointer authored
132 default, means an item never expires.
0c1b88b lots more docs.
Robey Pointer authored
133
58746d2 more docs!
Robey Pointer authored
134 Expired items are flushed from a queue whenever a new item is added or
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
135 removed. Additionally, if the global config option `expirationTimerFrequency`
136 is set, a background thread will periodically remove expired items from the
137 head of each queue. The provided `production.conf` sets this to one second.
138 If this is turned off, an idle queue won't have any items expired, but you
139 can still trigger a check by doing a "peek" on it.
140
141 Normally, expired items are discarded. If `expireToQueue` is set, then
142 expired items are moved to the specified queue just as if a client had put
143 it there. The item is added with no expiration time, but that can be
144 overridden if the new queue has a default expiration policy.
145
146 To prevent stalling the server when it encounters a swarm of items that all
147 expired at the same time, `maxExpireSweep` limits the number of items that
148 will be removed by the background thread in a single round. This is primarily
149 useful as a throttling mechanism when using a queue as a way to delay work.
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
150
7841145 Cory G Watson Expiring queues.
gphat authored
151 Queue expiration
152 ----------------
153
52c5fac Cory G Watson Modify various bit after review by robey.
gphat authored
154 Whole queues can be configured to expire as well. If `maxQueueAge` is set
155 `expirationTimerFrequency` is used to check the queue age. If the queue is
156 empty, and it has been longer than `maxQueueAge` since it was created then
157 the queue will be deleted.
7841145 Cory G Watson Expiring queues.
gphat authored
158
159 A `maxQueueAge` of zero, which is usually the default, means a queue never
160 expires.
0c1b88b lots more docs.
Robey Pointer authored
161
58746d2 more docs!
Robey Pointer authored
162 Fanout Queues
163 -------------
164
165 If a queue name has a `+` in it (like "`orders+audit`"), it's treated as a
166 fanout queue, using the format `<parent>+<child>`. These queues belong to a
167 parent queue -- in this example, the "orders" queue. Every item written into
168 a parent queue will also be written into each of its children.
169
170 Fanout queues each have their own journal file (if the parent queue has a
171 journal file) and otherwise behave exactly like any other queue. You can get
172 and peek and even add items directly to a child queue if you want. It uses the
173 parent queue's configuration instead of having independent child queue
174 configuration blocks.
175
176 When a fanout queue is first referenced by a client, the journal file (if any)
177 is created, and it will start receiving new items written to the parent queue.
178 Existing items are not copied over. A fanout queue can be deleted to stop it
179 from receiving new items.
180
181 Memcache commands
182 -----------------
183
184 - `SET <queue-name> <flags (ignored)> <expiration> <# bytes>`
185
186 Add an item to a queue. It may fail if the queue has a size or item limit
187 and it's full.
188
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
189 - `GET <queue-name>[options]`
66c498b docs!
Robey Pointer authored
190
58746d2 more docs!
Robey Pointer authored
191 Remove an item from a queue. It will return an empty response immediately if
192 the queue is empty. The queue name may be followed by options separated
193 by `/`:
194
66c498b docs!
Robey Pointer authored
195 - `/t=<milliseconds>`
196
197 Wait up to a given time limit for a new item to arrive. If an item arrives
198 on the queue within this timeout, it's returned as normal. Otherwise,
199 after that timeout, an empty response is returned.
200
201 - `/open`
202
203 Tentatively remove an item from the queue. The item is returned as usual
204 but is also set aside in case the client disappears before sending a
205 "close" request. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
206
66c498b docs!
Robey Pointer authored
207 - `/close`
58746d2 more docs!
Robey Pointer authored
208
66c498b docs!
Robey Pointer authored
209 Close any existing open read. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
210
66c498b docs!
Robey Pointer authored
211 - `/abort`
58746d2 more docs!
Robey Pointer authored
212
66c498b docs!
Robey Pointer authored
213 Cancel any existing open read, returing that item to the head of the
214 queue. It will be the next item fetched. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
215
66c498b docs!
Robey Pointer authored
216 - `/peek`
58746d2 more docs!
Robey Pointer authored
217
66c498b docs!
Robey Pointer authored
218 Return the first available item from the queue, if there is one, but don't
219 remove it. You can't combine this with any of the reliable read options.
58746d2 more docs!
Robey Pointer authored
220
d1b4ee7 R. Tyler Croy Update guide.md
rtyler authored
221 For example, to open a new read, waiting up to 500msec for an item:
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
222
223 GET work/t=500/open
224
225 Or to close an existing read and open a new one:
226
227 GET work/close/open
228
58746d2 more docs!
Robey Pointer authored
229 - `DELETE <queue-name>`
230
231 Drop a queue, discarding any items in it, and deleting any associated
232 journal files.
233
234 - `FLUSH <queue-name>`
235
236 Discard all items remaining in this queue. The queue remains live and new
237 items can be added. The time it takes to flush will be linear to the current
238 queue size, and any other activity on this queue will block while it's being
239 flushed.
240
241 - `FLUSH_ALL`
242
243 Discard all items remaining in all queues. The queues are flushed one at a
244 time, as if kestrel received a `FLUSH` command for each queue.
0c1b88b lots more docs.
Robey Pointer authored
245
58746d2 more docs!
Robey Pointer authored
246 - `VERSION`
247
248 Display the kestrel version in a way compatible with memcache.
249
250 - `SHUTDOWN`
251
252 Cleanly shutdown the server and exit.
253
254 - `RELOAD`
255
256 Reload the config file and reconfigure all queues. This should have no
257 noticable effect on the server's responsiveness.
258
259 - `DUMP_CONFIG`
260
261 Dump a list of each queue currently known to the server, and list the config
262 values for each queue. The format is:
263
66c498b docs!
Robey Pointer authored
264 queue 'master' {
265 max_items=2147483647
266 max_size=9223372036854775807
267 max_age=0
268 max_journal_size=16277216
269 max_memory_size=134217728
270 max_journal_overflow=10
271 max_journal_size_absolute=9223372036854775807
272 discard_old_when_full=false
273 journal=true
274 sync_journal=false
275 }
276
277 The last queue will be followed by `END` on a line by itself.
58746d2 more docs!
Robey Pointer authored
278
279 - `STATS`
280
281 Display server stats in memcache style. They're described below.
282
283 - `DUMP_STATS`
284
285 Display server stats in a more readable style, grouped by queue. They're
286 described below.
287
1531d0d finish cleaning up the guide.
Robey Pointer authored
288 - `MONITOR <queue-name> <seconds>`
289
290 Monitor a queue for a time, fetching any new items that arrive. Clients
291 are queued in a fair fashion, per-item, so many clients may monitor a
292 queue at once. After the given timeout, a separate `END` response will
293 signal the end of the monitor period. Any fetched items are open
294 transactions (see "Reliable Reads" below), and should be closed with
295 `CONFIRM`.
296
297 - `CONFIRM <queue-name> <count>`
298
299 Confirm receipt of `count` items from a queue. Usually this is the response
300 to a `MONITOR` command, to confirm the items that arrived during the monitor
301 period.
302
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
303
58746d2 more docs!
Robey Pointer authored
304 Reliable reads
305 --------------
306
307 Normally when a client removes an item from the queue, kestrel immediately
308 discards the item and assumes the client has taken ownership. This isn't
309 always safe, because a client could crash or lose the network connection
310 before it gets the item. So kestrel also supports a "reliable read" that
311 happens in two stages, using the `/open` and `/close` options to `GET`.
312
313 When `/open` is used, and an item is available, kestrel will remove it from
314 the queue and send it to the client as usual. But it will also set the item
315 aside. If a client disconnects while it has an open read, the item is put back
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
316 into the queue, at the head, so it will be the next item fetched. Only one
317 item can be "open" per client connection.
58746d2 more docs!
Robey Pointer authored
318
319 A previous open request is closed with `/close`. The server will reject any
320 attempt to open another read when one is already open, but it will ignore
321 `/close` if there's no open request, so that you can add `/close` to every
322 `GET` request for convenience.
323
324 If for some reason you want to abort a read without disconnecting, you can use
325 `/abort`. But because aborted items are placed back at the head of the queue,
326 this isn't a good way to deal with client errors. Since the error-causing item
327 will always be the next one available, you'll end up bouncing the same item
328 around between clients instead of making progress.
329
330 There's always a trade-off: either potentially lose items or potentially
331 receive the same item multiple times. Reliable reads choose the latter option.
332 To use this tactic successfully, work items should be idempotent, meaning the
333 work could be done 2 or 3 times and have the same effect as if it had been
334 done only once (except wasting some resources).
335
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
336 Example:
337
338 GET dirty_jobs/close/open
339 (receives job 1)
340 GET dirty_jobs/close/open
341 (closes job 1, receives job 2)
342 ...etc...
58746d2 more docs!
Robey Pointer authored
343
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
344
58746d2 more docs!
Robey Pointer authored
345 Server stats
346 ------------
347
66c498b docs!
Robey Pointer authored
348 Global stats reported by kestrel are:
349
350 - `uptime` - seconds the server has been online
351 - `time` - current time in unix epoch
352 - `version` - version string, like "1.2"
353 - `curr_items` - total of items waiting in all queues
354 - `total_itmes` - total of items that have ever been added in this server's
355 lifetime
356 - `bytes` - total byte size of items waiting in all queues
357 - `curr_connections` - current open connections from clients
358 - `total_connections` - total connections that have been opened in this
359 server's lifetime
360 - `cmd_get` - total `GET` requests
361 - `cmd_set` - total `SET` requests
362 - `cmd_peek` - total `GET/peek` requests
363 - `get_hits` - total `GET` requests that received an item
364 - `get_misses` - total `GET` requests on an empty queue
365 - `bytes_read` - total bytes read from clients
366 - `bytes_written` - total bytes written to clients
0d8a507 Cory G Watson Add more statistics along with tests and docs.
gphat authored
367 - `queue_creates` - total number of queues created
7841145 Cory G Watson Expiring queues.
gphat authored
368 - `queue_deletes` - total number of queues deleted (includes expires)
369 - `queue_expires` - total number of queues expires
66c498b docs!
Robey Pointer authored
370
371 For each queue, the following stats are also reported:
372
373 - `items` - items waiting in this queue
374 - `bytes` - total byte size of items waiting in this queue
375 - `total_items` - total items that have been added to this queue in this
376 server's lifetime
377 - `logsize` - byte size of the queue's journal file
378 - `expired_items` - total items that have been expired from this queue in this
379 server's lifetime
380 - `mem_items` - items in this queue that are currently in memory
381 - `mem_bytes` - total byte size of items in this queue that are currently in
382 memory (will always be less than or equal to `max_memory_size` config for
383 the queue)
384 - `age` - time, in milliseconds, that the last item to be fetched from this
385 queue had been waiting; that is, the time between `SET` and `GET`; if the
386 queue is empty, this will always be zero
387 - `discarded` - number of items discarded because the queue was too full
388 - `waiters` - number of clients waiting for an item from this queue (using
389 `GET/t`)
721bf6b document the new stat.
Robey Pointer authored
390 - `open_transactions` - items read with `/open` but not yet confirmed
0d8a507 Cory G Watson Add more statistics along with tests and docs.
gphat authored
391 - `total_flushes` total number of times this queue has been flushed
66c498b docs!
Robey Pointer authored
392
393
394 Kestrel as a library
395 --------------------
396
397 You can use kestrel as a library by just sticking the jar on your classpath.
398 It's a cheap way to get a durable work queue for inter-process or inter-thread
399 communication. Each queue is represented by a `PersistentQueue` object:
400
1531d0d finish cleaning up the guide.
Robey Pointer authored
401 class PersistentQueue(val name: String, persistencePath: String,
402 @volatile var config: QueueConfig, timer: Timer,
403 queueLookup: Option[(String => Option[PersistentQueue])]) {
66c498b docs!
Robey Pointer authored
404
405 and must be initialized before using:
406
407 def setup(): Unit
58746d2 more docs!
Robey Pointer authored
408
66c498b docs!
Robey Pointer authored
409 specifying the path for the journal files (if the queue will be journaled),
1531d0d finish cleaning up the guide.
Robey Pointer authored
410 the name of the queue, a `QueueConfig` object (derived from `QueueBuilder`),
411 a timer for handling timeout reads, and optionally a way to find other named
412 queues (for `expireToQueue` support).
58746d2 more docs!
Robey Pointer authored
413
66c498b docs!
Robey Pointer authored
414 To add an item to a queue:
58746d2 more docs!
Robey Pointer authored
415
1531d0d finish cleaning up the guide.
Robey Pointer authored
416 def add(value: Array[Byte], expiry: Option[Time]): Boolean
58746d2 more docs!
Robey Pointer authored
417
66c498b docs!
Robey Pointer authored
418 It will return `false` if the item was rejected because the queue was full.
0c1b88b lots more docs.
Robey Pointer authored
419
66c498b docs!
Robey Pointer authored
420 Queue items are represented by a case class:
421
1531d0d finish cleaning up the guide.
Robey Pointer authored
422 case class QItem(addTime: Time, expiry: Option[Time], data: Array[Byte], var xid: Int)
66c498b docs!
Robey Pointer authored
423
424 and several operations exist to remove or peek at the head item:
425
426 def peek(): Option[QItem]
427 def remove(): Option[QItem]
428
429 To open a reliable read, set `transaction` true, and later confirm or unremove
430 the item by its `xid`:
431
432 def remove(transaction: Boolean): Option[QItem]
433 def unremove(xid: Int)
434 def confirmRemove(xid: Int)
435
1531d0d finish cleaning up the guide.
Robey Pointer authored
436 You can also asynchronously remove or peek at items using futures.
66c498b docs!
Robey Pointer authored
437
1531d0d finish cleaning up the guide.
Robey Pointer authored
438 def waitRemove(deadline: Option[Time], transaction: Boolean): Future[Option[QItem]]
439 def waitPeek(deadline: Option[Time]): Future[Option[QItem]]
66c498b docs!
Robey Pointer authored
440
441 When done, you should close the queue:
442
443 def close(): Unit
444 def isClosed: Boolean
445
446 Here's a short example:
447
1531d0d finish cleaning up the guide.
Robey Pointer authored
448 var queue = new PersistentQueue("work", "/var/spool/kestrel", config, timer, None)
66c498b docs!
Robey Pointer authored
449 queue.setup()
450
451 // add an item with no expiration:
452 queue.add("hello".getBytes, 0)
453
454 // start to remove it, then back out:
455 val item = queue.remove(true)
456 queue.unremove(item.xid)
457
458 // remove an item with a 500msec timeout, and confirm it:
1531d0d finish cleaning up the guide.
Robey Pointer authored
459 queue.waitRemove(500.milliseconds.fromNow, true)() match {
460 case None =>
461 println("nothing. :(")
462 case Some(item) =>
463 println("got: " + new String(item.data))
464 queue.confirmRemove(item.xid)
66c498b docs!
Robey Pointer authored
465 }
0c1b88b lots more docs.
Robey Pointer authored
466
66c498b docs!
Robey Pointer authored
467 queue.close()
Something went wrong with that request. Please try again.