Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 665 lines (491 sloc) 28.169 kB
0c1b88b lots more docs.
Robey Pointer authored
1
62482e2 cleanup.
Robey Pointer authored
2 A working guide to kestrel
3 ==========================
0c1b88b lots more docs.
Robey Pointer authored
4
1ae7657 kestrel: server status & zookeeper server sets
Stephan Zuercher authored
5 Kestrel is a very simple message queue that runs on the JVM. It supports
6 multiple protocols:
e224a6f minor docs/guide.md update
Stephan Zuercher authored
7
8 - memcache: the memcache protocol, with some extensions
9 - thrift: Apache Thrift-based RPC
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
10 - text: a simple text-based protocol
0c1b88b lots more docs.
Robey Pointer authored
11
12 A single kestrel server has a set of queues identified by a name, which is
13 also the filename of that queue's journal file (usually in
14 `/var/spool/kestrel`). Each queue is a strictly-ordered FIFO of "items" of
15 binary data. Usually this data is in some serialized format like JSON or
16 ruby's marshal format.
17
d6199a9 explain queue names.
Robey Pointer authored
18 Generally queue names should be limited to alphanumerics `[A-Za-z0-9]`, dash
19 (`-`) and underline (`_`). In practice, kestrel doesn't enforce any
20 restrictions other than the name can't contain slash (`/`) because that can't
21 be used in filenames, squiggle (`~`) because it's used for temporary files,
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
22 plus (`+`) because it's used for fanout queues, and dot (`.`) because it's
23 reserved for future use. Queue names are case-sensitive, but if you're running
24 kestrel on OS X or Windows, you will want to refrain from taking advantage of
25 this, since the journal filenames on those two platforms are *not*
26 case-sensitive.
d6199a9 explain queue names.
Robey Pointer authored
27
0c1b88b lots more docs.
Robey Pointer authored
28 A cluster of kestrel servers is like a memcache cluster: the servers don't
29 know about each other, and don't do any cross-communication, so you can add as
1ae7657 kestrel: server status & zookeeper server sets
Stephan Zuercher authored
30 many as you like. The simplest clients have a list of all servers in the
31 cluster, and pick one at random for each operation. In this way, each queue
32 appears to be spread out across every server, with items in a loose ordering.
33 More advanced clients can find kestrel servers via ZooKeeper.
0c1b88b lots more docs.
Robey Pointer authored
34
58746d2 more docs!
Robey Pointer authored
35 When kestrel starts up, it scans the journal folder and creates queues based
36 on any journal files it finds there, to restore state to the way it was when
37 it last shutdown (or was killed or died). New queues are created by referring
38 to them (for example, adding or trying to remove an item). A queue can be
39 deleted with the "delete" command.
40
0c1b88b lots more docs.
Robey Pointer authored
41
42 Configuration
43 -------------
44
bf4ac4d KEST-272: proper config inheritance; log queue/alias config on reload
Stephan Zuercher authored
45 **NOTE:** Kestrel 2.3.4 introduces inheritance for queue configurations. For more
46 information, see below.
47
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
48 The config files for kestrel are scala expressions loaded at runtime, usually
49 from `production.scala`, although you can use `development.scala` by passing
50 `-Dstage=development` to the java command line.
51
52 The config file evaluates to a `KestrelConfig` object that's used to configure
53 the server as a whole, a default queue, and any overrides for specific named
54 queues. The fields on `KestrelConfig` are documented here with their default
55 values:
bf4ac4d KEST-272: proper config inheritance; log queue/alias config on reload
Stephan Zuercher authored
56 [KestrelConfig](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/KestrelConfig.html)
0c1b88b lots more docs.
Robey Pointer authored
57
58 To confirm the current configuration of each queue, send "dump_config" to
59 a server (which can be done over telnet).
60
61 To reload the config file on a running server, send "reload" the same way.
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
62 You should immediately see the changes in "dump_config", to confirm. Reloading
1ae7657 kestrel: server status & zookeeper server sets
Stephan Zuercher authored
63 will only affect queue and alias configuration, not global server configuration.
64 To change the server configuration, restart the server.
0c1b88b lots more docs.
Robey Pointer authored
65
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
66 Logging is configured according to `util-logging`. The logging configuration
67 syntax is described here:
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
68 [util-logging](https://github.com/twitter/util/blob/master/util-logging/README.markdown)
0c1b88b lots more docs.
Robey Pointer authored
69
bf4ac4d KEST-272: proper config inheritance; log queue/alias config on reload
Stephan Zuercher authored
70 Per-queue configuration options are documented here:
71 [QueueBuilder](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/QueueBuilder.html)
72
73 Queue alias configuration options are documented here:
74 [AliasBuilder](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/AliasBuilder.html)
75
76 Configuration Changes Starting in Kestrel 2.3.4
77 -----------------------------------------------
78
79 Starting with Kestrel 2.3.4, queue configurations are inherited:
80
81 * Any queue with no explict configuration (see `queues` in `KestrelConfig`) uses the default
82 queue configuration (see `default` in `KestrelConfig`). This behavior is unchanged from
83 previous versions.
9ffcd5c minor guide edit
Stephan Zuercher authored
84 * Any master (e.g. not fanout) queue with an explicit queue configuration overrides the default
85 queue configuration. For example, if `default.maxMemorySize` is set, all explicitly configured
bf4ac4d KEST-272: proper config inheritance; log queue/alias config on reload
Stephan Zuercher authored
86 queues will inherit that setting *unless* explicitly overridden in the queue's configuration.
87 Older versions of Kestrel *did not* apply values from the default queue configuration to any
88 explicitly configured queue.
89 * Any fanout queue (e.g., a queue with a `+` in its name), inherits its master queue's
90 configuration, unless explicitly overridden (see `queues` in `KestrelConfig`). Older versions
9ffcd5c minor guide edit
Stephan Zuercher authored
91 of Kestrel silently ignored explicit fanout queue configurations and used the master queue's
92 configuration.
bf4ac4d KEST-272: proper config inheritance; log queue/alias config on reload
Stephan Zuercher authored
93
94 ### Example Configuration
95 -------------------------
96
97 Existing configurations should continue to load, but the resulting configuration may
98 differ. As an example, the following configuration file and table illustrate the differences
99 between a configuration loaded by Kestrel 2.3.3 and Kestrel 2.3.4 (and later).
100
101 new KestrelConfig {
102 default.maxMemorySize = 8.megabytes
103
104 queues = new QueueBuilder() {
105 name = "q"
106 maxItems = 500
107 } :: new QueueBuilder() {
108 name = "q+fanout"
109 maxAge = 1.minute
110 } :: new QueueBuilder() {
111 name = "x"
112 maxMemorySize = 16.megabytes
113 }
114 }
115
116
117 <table>
fcabfac fix angle brackets
Stephan Zuercher authored
118 <tr><th>Queue</th> <th>Setting</th> <th>Kestrel &lt;= 2.3.3</th> <th>Kestrel &gt;= 2.3.4</th> </tr>
bf4ac4d KEST-272: proper config inheritance; log queue/alias config on reload
Stephan Zuercher authored
119 <tr><td>q</td> <td>maxMemorySize</td> <td>128.megabytes</td> <td>8.megabytes</td> </tr>
120 <tr><td>q+fanout</td> <td>maxMemorySize</td> <td>128.megabytes</td> <td>8.megabytes</td> </tr>
121 <tr><td>x</td> <td>maxMemorySize</td> <td>16.megabytes</td> <td>16.megabytes</td> </tr>
122 <tr><td>q</td> <td>maxItems</td> <td>500</td> <td>500</td> </tr>
123 <tr><td>q+fanout</td> <td>maxItems</td> <td>500</td> <td>500</td> </tr>
124 <tr><td>q+fanout</td> <td>maxAge</td> <td>None</td> <td>Some(1.minute)</td> </tr>
125 </table>
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
126
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
127
128 Full queues
129 -----------
130
131 A queue can have the following limits set on it:
132
133 - `maxItems` - total items in the queue
134 - `maxSize` - total bytes of data in the items in the queue
135
136 If either of these limits is reached, no new items can be added to the queue.
137 (Clients will receive an error when trying to add.) If you set
138 `discardOldWhenFull` to true, then all adds will succeed, and the oldest
139 item(s) will be silently discarded until the queue is back within the item
140 and size limits.
141
142 `maxItemSize` limits the size of any individual item. If an add is attempted
143 with an item larger than this limit, it always fails.
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_…
Robey Pointer authored
144
0c1b88b lots more docs.
Robey Pointer authored
145
146 The journal file
147 ----------------
148
149 The journal file is the only on-disk storage of a queue's contents, and it's
150 just a sequential record of each add or remove operation that's happened on
151 that queue. When kestrel starts up, it replays each queue's journal to build
152 up the in-memory queue that it uses for client queries.
153
f3f5a4a never rewrite journals unless the queue is empty or in read-behind
Stephan Zuercher authored
154 The journal file is compacted if the queue is empty and the journal is larger
155 than `defaultJournalSize`.
0c1b88b lots more docs.
Robey Pointer authored
156
f3f5a4a never rewrite journals unless the queue is empty or in read-behind
Stephan Zuercher authored
157 The current journal file is archived (or rotated) when the journal is larger
158 than `maxMemorySize`. In addition, if the complete journal exceeds
159 `maxJournalSize`, a checkpoint is set. The journal may then be compacted during
160 read-behind (when the size of the queue exceeds `maxMemorySize`) via the
161 journal packer thread.
0c1b88b lots more docs.
Robey Pointer authored
162
1531d0d finish cleaning up the guide.
Robey Pointer authored
163 For example, if `defaultJournalSize` is 16MB (the default), then if the queue
f3f5a4a never rewrite journals unless the queue is empty or in read-behind
Stephan Zuercher authored
164 is empty and the journal is larger than 16MB, it will be compacted into a new
165 (empty, if there are no open transactions) file.
0c1b88b lots more docs.
Robey Pointer authored
166
1531d0d finish cleaning up the guide.
Robey Pointer authored
167 You can turn the journal off for a queue (`keepJournal` = false) and the queue
0c1b88b lots more docs.
Robey Pointer authored
168 will exist only in memory. If the server restarts, all enqueued items are
1531d0d finish cleaning up the guide.
Robey Pointer authored
169 lost. You can also force a queue's journal to be sync'd to disk periodically,
170 or even after every write operation, at a performance cost, using
171 `syncJournal`.
0c1b88b lots more docs.
Robey Pointer authored
172
1531d0d finish cleaning up the guide.
Robey Pointer authored
173 If a queue grows past `maxMemorySize` bytes (128MB by default), only the
0c1b88b lots more docs.
Robey Pointer authored
174 first 128MB is kept in memory. The journal is used to track later items, and
175 as items are removed, the journal is played forward to keep 128MB in memory.
176 This is usually known as "read-behind" mode, but Twitter engineers sometimes
177 refer to it as the "square snake" because of the diagram used to brainstorm
178 the implementation. When a queue is in read-behind mode, removing an item will
179 often cause 2 disk operations instead of one: one to record the remove, and
180 one to read an item in from disk to keep 128MB in memory. This is the
181 trade-off to avoid filling memory and crashing the JVM.
182
183
58746d2 more docs!
Robey Pointer authored
184 Item expiration
185 ---------------
0c1b88b lots more docs.
Robey Pointer authored
186
58746d2 more docs!
Robey Pointer authored
187 When they come from a client, expiration times are handled in the same way as
188 memcache: if the number is small (less than one million), it's interpreted as
189 a relative number of seconds from now. Otherwise it's interpreted as an
190 absolute unix epoch time, in seconds since the beginning of 1 January 1970
191 GMT.
0c1b88b lots more docs.
Robey Pointer authored
192
58746d2 more docs!
Robey Pointer authored
193 Expiration times are immediately translated into an absolute time, in
1531d0d finish cleaning up the guide.
Robey Pointer authored
194 *milliseconds*, and if it's further in the future than the queue's `maxAge`,
a3c112f fix typos
Robey Pointer authored
195 the `maxAge` is used instead. An expiration of 0, which is usually the
58746d2 more docs!
Robey Pointer authored
196 default, means an item never expires.
0c1b88b lots more docs.
Robey Pointer authored
197
58746d2 more docs!
Robey Pointer authored
198 Expired items are flushed from a queue whenever a new item is added or
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
199 removed. Additionally, if the global config option `expirationTimerFrequency`
200 is set, a background thread will periodically remove expired items from the
201 head of each queue. The provided `production.conf` sets this to one second.
202 If this is turned off, an idle queue won't have any items expired, but you
203 can still trigger a check by doing a "peek" on it.
204
205 Normally, expired items are discarded. If `expireToQueue` is set, then
206 expired items are moved to the specified queue just as if a client had put
207 it there. The item is added with no expiration time, but that can be
208 overridden if the new queue has a default expiration policy.
209
210 To prevent stalling the server when it encounters a swarm of items that all
211 expired at the same time, `maxExpireSweep` limits the number of items that
212 will be removed by the background thread in a single round. This is primarily
213 useful as a throttling mechanism when using a queue as a way to delay work.
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_…
Robey Pointer authored
214
7841145 @gphat Expiring queues.
gphat authored
215 Queue expiration
216 ----------------
217
e224a6f minor docs/guide.md update
Stephan Zuercher authored
218 Whole queues can be configured to expire as well. If `maxQueueAge` is set
52c5fac @gphat Modify various bit after review by robey.
gphat authored
219 `expirationTimerFrequency` is used to check the queue age. If the queue is
220 empty, and it has been longer than `maxQueueAge` since it was created then
221 the queue will be deleted.
0c1b88b lots more docs.
Robey Pointer authored
222
e224a6f minor docs/guide.md update
Stephan Zuercher authored
223
58746d2 more docs!
Robey Pointer authored
224 Fanout Queues
225 -------------
226
227 If a queue name has a `+` in it (like "`orders+audit`"), it's treated as a
228 fanout queue, using the format `<parent>+<child>`. These queues belong to a
229 parent queue -- in this example, the "orders" queue. Every item written into
230 a parent queue will also be written into each of its children.
231
232 Fanout queues each have their own journal file (if the parent queue has a
233 journal file) and otherwise behave exactly like any other queue. You can get
234 and peek and even add items directly to a child queue if you want. It uses the
235 parent queue's configuration instead of having independent child queue
236 configuration blocks.
237
238 When a fanout queue is first referenced by a client, the journal file (if any)
239 is created, and it will start receiving new items written to the parent queue.
240 Existing items are not copied over. A fanout queue can be deleted to stop it
241 from receiving new items.
242
6993502 Document fanoutOnly.
Cory Watson authored
243 `fanoutOnly` may be set to true if the queue in question will only serve write
244 point for fanout queues. No journal file will be kept for the parent, only
245 for the child queues. This saves the overhead of writing to the parent and
246 removes the need to empty it. Note that setting `fanoutOnly` to true and
247 having no fanouts for the queue effectively makes it a black hole.
e224a6f minor docs/guide.md update
Stephan Zuercher authored
248
ff976b0 introduce queue aliases (including stats)
Stephan Zuercher authored
249 Queue Aliases
250 -------------
251
252 Queue aliases are somewhat similar to fanout queues, but without a required
253 naming convention or implicit creation of child queues. A queue alias can
254 only be used in set operations. Kestrel responds to attempts to retrieve
255 items from the alias as if it were an empty queue. Delete and flush requests
256 are also ignored.
58746d2 more docs!
Robey Pointer authored
257
e224a6f minor docs/guide.md update
Stephan Zuercher authored
258
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
259 Protocols
260 ---------
261
262 Kestrel supports three protocols: memcache, thrift and text. The
263 [Finagle project](http://twitter.github.com/finagle/) can be used to connect clients
264 to a Kestrel server via the memcache or thrift protocols.
bb7eb5e rename xid to id in the thrift interface, and link it from the docs
Robey Pointer authored
265
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
266 ### Thrift
267 ----------
268
269 The thrift protocol is documented in the thrift IDL:
bb7eb5e rename xid to id in the thrift interface, and link it from the docs
Robey Pointer authored
270 [kestrel.thrift](https://github.com/robey/kestrel/blob/master/src/main/thrift/kestrel.thrift)
271
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
272 Reliable reads via the thrift protocol are specified by indicating how long the server
273 should wait before aborting the unacknowledged read.
bb7eb5e rename xid to id in the thrift interface, and link it from the docs
Robey Pointer authored
274
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
275
276 ### Memcache
277 ------------
278
279 The official memcache protocol is described here:
280 [protocol.txt](https://github.com/memcached/memcached/blob/master/doc/protocol.txt)
281
282 The kestrel implementation of the memcache protocol commands is described below.
58746d2 more docs!
Robey Pointer authored
283
284 - `SET <queue-name> <flags (ignored)> <expiration> <# bytes>`
285
286 Add an item to a queue. It may fail if the queue has a size or item limit
287 and it's full.
288
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
289 - `GET <queue-name>[options]`
66c498b docs!
Robey Pointer authored
290
58746d2 more docs!
Robey Pointer authored
291 Remove an item from a queue. It will return an empty response immediately if
292 the queue is empty. The queue name may be followed by options separated
293 by `/`:
294
66c498b docs!
Robey Pointer authored
295 - `/t=<milliseconds>`
296
297 Wait up to a given time limit for a new item to arrive. If an item arrives
298 on the queue within this timeout, it's returned as normal. Otherwise,
299 after that timeout, an empty response is returned.
300
301 - `/open`
302
303 Tentatively remove an item from the queue. The item is returned as usual
304 but is also set aside in case the client disappears before sending a
305 "close" request. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
306
66c498b docs!
Robey Pointer authored
307 - `/close`
58746d2 more docs!
Robey Pointer authored
308
66c498b docs!
Robey Pointer authored
309 Close any existing open read. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
310
66c498b docs!
Robey Pointer authored
311 - `/abort`
58746d2 more docs!
Robey Pointer authored
312
66c498b docs!
Robey Pointer authored
313 Cancel any existing open read, returing that item to the head of the
314 queue. It will be the next item fetched. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
315
66c498b docs!
Robey Pointer authored
316 - `/peek`
58746d2 more docs!
Robey Pointer authored
317
66c498b docs!
Robey Pointer authored
318 Return the first available item from the queue, if there is one, but don't
319 remove it. You can't combine this with any of the reliable read options.
58746d2 more docs!
Robey Pointer authored
320
d1b4ee7 @rtyler Update guide.md
rtyler authored
321 For example, to open a new read, waiting up to 500msec for an item:
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
322
323 GET work/t=500/open
324
325 Or to close an existing read and open a new one:
326
327 GET work/close/open
328
58746d2 more docs!
Robey Pointer authored
329 - `DELETE <queue-name>`
330
331 Drop a queue, discarding any items in it, and deleting any associated
332 journal files.
333
334 - `FLUSH <queue-name>`
335
336 Discard all items remaining in this queue. The queue remains live and new
337 items can be added. The time it takes to flush will be linear to the current
338 queue size, and any other activity on this queue will block while it's being
339 flushed.
340
341 - `FLUSH_ALL`
342
343 Discard all items remaining in all queues. The queues are flushed one at a
344 time, as if kestrel received a `FLUSH` command for each queue.
0c1b88b lots more docs.
Robey Pointer authored
345
58746d2 more docs!
Robey Pointer authored
346 - `VERSION`
347
348 Display the kestrel version in a way compatible with memcache.
349
350 - `SHUTDOWN`
351
352 Cleanly shutdown the server and exit.
353
354 - `RELOAD`
355
356 Reload the config file and reconfigure all queues. This should have no
357 noticable effect on the server's responsiveness.
358
359 - `STATS`
360
361 Display server stats in memcache style. They're described below.
362
363 - `DUMP_STATS`
364
365 Display server stats in a more readable style, grouped by queue. They're
366 described below.
367
55c7342 allow memcache protocol to have an optional max_items on "monitor".
Robey Pointer authored
368 - `MONITOR <queue-name> <seconds> [max-items]`
1531d0d finish cleaning up the guide.
Robey Pointer authored
369
e224a6f minor docs/guide.md update
Stephan Zuercher authored
370 Monitor a queue for a time, fetching any new items that arrive, up to an
371 optional maximum number of items. Clients are queued in a fair fashion,
372 per-item, so many clients may monitor a queue at once. After the given
373 timeout, a separate `END` response will signal the end of the monitor
374 period. Any fetched items are open transactions (see "Reliable Reads"
59e3d33 modernize kestrel libraries
Stephan Zuercher authored
375 below), and should be closed with `CONFIRM`.
1531d0d finish cleaning up the guide.
Robey Pointer authored
376
377 - `CONFIRM <queue-name> <count>`
378
379 Confirm receipt of `count` items from a queue. Usually this is the response
380 to a `MONITOR` command, to confirm the items that arrived during the monitor
381 period.
382
1ae7657 kestrel: server status & zookeeper server sets
Stephan Zuercher authored
383 - `STATUS`
384
385 Displays the kestrel server's current status (see section on Server Status,
386 below).
387
388 - `STATUS <new-status>`
389
390 Switches the kestrel server's current status to the given status (see section
391 on Server Status, below).
392
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_…
Robey Pointer authored
393
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
394 #### Reliable reads
395 -------------------
58746d2 more docs!
Robey Pointer authored
396
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
397 Note: this section is specific to the memcache protocol.
e224a6f minor docs/guide.md update
Stephan Zuercher authored
398
58746d2 more docs!
Robey Pointer authored
399 Normally when a client removes an item from the queue, kestrel immediately
400 discards the item and assumes the client has taken ownership. This isn't
401 always safe, because a client could crash or lose the network connection
402 before it gets the item. So kestrel also supports a "reliable read" that
403 happens in two stages, using the `/open` and `/close` options to `GET`.
404
405 When `/open` is used, and an item is available, kestrel will remove it from
406 the queue and send it to the client as usual. But it will also set the item
407 aside. If a client disconnects while it has an open read, the item is put back
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
408 into the queue, at the head, so it will be the next item fetched. Only one
409 item can be "open" per client connection.
58746d2 more docs!
Robey Pointer authored
410
411 A previous open request is closed with `/close`. The server will reject any
412 attempt to open another read when one is already open, but it will ignore
413 `/close` if there's no open request, so that you can add `/close` to every
414 `GET` request for convenience.
415
416 If for some reason you want to abort a read without disconnecting, you can use
417 `/abort`. But because aborted items are placed back at the head of the queue,
418 this isn't a good way to deal with client errors. Since the error-causing item
419 will always be the next one available, you'll end up bouncing the same item
420 around between clients instead of making progress.
421
422 There's always a trade-off: either potentially lose items or potentially
423 receive the same item multiple times. Reliable reads choose the latter option.
424 To use this tactic successfully, work items should be idempotent, meaning the
425 work could be done 2 or 3 times and have the same effect as if it had been
426 done only once (except wasting some resources).
427
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
428 Example:
429
430 GET dirty_jobs/close/open
431 (receives job 1)
432 GET dirty_jobs/close/open
433 (closes job 1, receives job 2)
434 ...etc...
58746d2 more docs!
Robey Pointer authored
435
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_…
Robey Pointer authored
436
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
437 ### Text protocol
438 -----------------
439
440 Kestrel supports a limited, text-only protocol. You are encouraged to use the
441 memcache protocol instead.
442
443 The text protocol does not support reliable reads.
444
445
1ae7657 kestrel: server status & zookeeper server sets
Stephan Zuercher authored
446 Server Status
447 -------------
448
449 Each kestrel server maintains its current status. Normal statuses are
450
451 - `Up`: the server is available for all operations
452 - `ReadOnly`: the server is available for non-modifying operations only;
453 commands that modify queues (set, delete, flush) are rejected as
454 errors.
455 - `Quiescent`: the server rejects as an error operations on any queue. One
456 notable exception is transactions begun before the server entered
457 the quiesecent state may still be confirmed.
458
459 One additional status is `Down`, which is only used transiently when kestrel is
460 in the process of shutting down.
461
462 The server's current status is persisted (specified in
463 [KestrelConfig](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/KestrelConfig.html)).
464 When kestrel is restarted it automatically returns to it's previous status,
465 based on the value in the status file. If the status file does not exist or
466 cannot be read, kestrel uses a default status, also configured in KestrelConfig.
467
468 When changing from a less restrictive status to a more restrictive status
469 (e.g., from `Up` to `ReadOnly` or from `ReadOnly` to `Quiescent`), the
470 config option `statusChangeGracePeriod` determines how long kestrel will
471 continue to allow restricted operations to continue before it begins rejecting
472 them. This allows clients that are aware of the kestrel server's status a
473 grace period to learn the new status and cease the forbidden operations before
474 beginning to encounter errors.
475
476 ### ZooKeeper Server Sets
477 -------------------------
478
94bae3e better lookup of external IP address for ZK endpoints; don't advertis…
Stephan Zuercher authored
479 Kestrel uses Twitter's ServerSet library to support client discovery of kestrel
1ae7657 kestrel: server status & zookeeper server sets
Stephan Zuercher authored
480 servers allowing a given operation. The ServerSet class is documented here:
481 [ServerSet](http://twitter.github.com/commons/apidocs/index.html#com.twitter.common.zookeeper.ServerSet)
482
483 If the optional `zookeeper` field of `KestrelConfig` is specified, kestrel will
484 attempt to use the given configuration to join a logical set of kestrel servers.
485 The ZooKeeper host, port and other connection options are documented here:
486 [ZooKeeperBuilder](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/ZooKeeperBuilder.html)
487
488 Kestrel servers will join 0, 1, or 2 server sets depending on their current
489 status. When `Up`, the server joins two server sets: one for writes and one for
490 reads. When `ReadOnly`, the server joins only the read set. When `Quiescent`,
491 the server joins no sets. ZooKeeper-aware kestrel clients can watch the
492 server set for changes and adjust their connections accordingly. The
493 `statusChangeGracePeriod` configuration option may be used to allow clients
494 time to detect and react to the status change before they begin receiving
495 errors from kestrel.
496
94bae3e better lookup of external IP address for ZK endpoints; don't advertis…
Stephan Zuercher authored
497 The ZooKeeper path used to register the server set is based on the `pathPrefix`
498 option. Kestrel automatically appends `/write` and `/read` to distinguish the
499 write and read sets.
1ae7657 kestrel: server status & zookeeper server sets
Stephan Zuercher authored
500
501 Kestrel advertises all of its endpoints in each server set that it joins.
502 The default endpoint is memcache, if configured. The default endpoint falls
503 back to the thrift endpoint and then the text protocol endpoint. All three
504 endpoints are advertised as additional endpoints under the names `memcache`,
505 `thrift` and `text`.
506
94bae3e better lookup of external IP address for ZK endpoints; don't advertis…
Stephan Zuercher authored
507 Kestrel advertises only a single IP address per endpoint. This IP address is
508 based on Kestrel's `listenAddress`. If the listener address is the wildcard
509 address (e.g., `0.0.0.0` or `::`), Kestrel will advertise the first IP address
510 it finds by traversing the host's configured network interfaces (via
511 `java.net.NetworkInterface`). If your host has multiple, valid external IP
512 addresses you can choose the advertised address by setting the listener address
513 to that IP. Finally, If the listener address is a loopback address (e.g.,
514 `127.0.0.1` or `::1`), Kestrel will not start, since advertising a loopback
515 address on ZooKeeper will not work and no external host could connect to
516 Kestrel in any event.
517
1ae7657 kestrel: server status & zookeeper server sets
Stephan Zuercher authored
518 Consider setting the `defaultStatus` option to `Quiescent` to prevent kestrel
519 from prematurely advertising its status via ZooKeeper.
520
521 Installations that require additional customization of ZooKeeper credentials,
522 or other site-specific ZooKeeper initialization can override the
523 `clientInitializer` and `serverSetInitializer` options to invoke the
524 necessary site-specific code. The recommended implementation is to place
525 the site-specific code in its own JAR file, take the necessary steps to
526 include the JAR in kestrel's class path, and place as little logic as possible
527 in the kestrel configuration file.
528
529
58746d2 more docs!
Robey Pointer authored
530 Server stats
531 ------------
532
66c498b docs!
Robey Pointer authored
533 Global stats reported by kestrel are:
534
535 - `uptime` - seconds the server has been online
536 - `time` - current time in unix epoch
537 - `version` - version string, like "1.2"
538 - `curr_items` - total of items waiting in all queues
539 - `total_itmes` - total of items that have ever been added in this server's
540 lifetime
541 - `bytes` - total byte size of items waiting in all queues
542 - `curr_connections` - current open connections from clients
543 - `total_connections` - total connections that have been opened in this
544 server's lifetime
545 - `cmd_get` - total `GET` requests
546 - `cmd_set` - total `SET` requests
547 - `cmd_peek` - total `GET/peek` requests
548 - `get_hits` - total `GET` requests that received an item
549 - `get_misses` - total `GET` requests on an empty queue
550 - `bytes_read` - total bytes read from clients
551 - `bytes_written` - total bytes written to clients
0d8a507 @gphat Add more statistics along with tests and docs.
gphat authored
552 - `queue_creates` - total number of queues created
7841145 @gphat Expiring queues.
gphat authored
553 - `queue_deletes` - total number of queues deleted (includes expires)
554 - `queue_expires` - total number of queues expires
66c498b docs!
Robey Pointer authored
555
556 For each queue, the following stats are also reported:
557
558 - `items` - items waiting in this queue
559 - `bytes` - total byte size of items waiting in this queue
560 - `total_items` - total items that have been added to this queue in this
561 server's lifetime
562 - `logsize` - byte size of the queue's journal file
563 - `expired_items` - total items that have been expired from this queue in this
564 server's lifetime
565 - `mem_items` - items in this queue that are currently in memory
566 - `mem_bytes` - total byte size of items in this queue that are currently in
567 memory (will always be less than or equal to `max_memory_size` config for
568 the queue)
569 - `age` - time, in milliseconds, that the last item to be fetched from this
570 queue had been waiting; that is, the time between `SET` and `GET`; if the
571 queue is empty, this will always be zero
572 - `discarded` - number of items discarded because the queue was too full
573 - `waiters` - number of clients waiting for an item from this queue (using
574 `GET/t`)
721bf6b document the new stat.
Robey Pointer authored
575 - `open_transactions` - items read with `/open` but not yet confirmed
9afe5c4 track number of transactional get requests and cancellations per queue
Stephan Zuercher authored
576 - `transactions` - number of transactional get requests (irrespective of whether an
577 item was read or not)
578 - `canceled_transactions` - number of transactional get requests canceled (for any
579 reason)
5e4a681 @gphat Add create_time as a queue stat and document age_msec.
gphat authored
580 - `total_flushes` - total number of times this queue has been flushed
581 - `age_msec` - age of the last item read from the queue
582 - `create_time` - the time that the queue was created (in milliseconds since epoch)
66c498b docs!
Robey Pointer authored
583
c66bbb3 more docs/guide.md cleanup
Stephan Zuercher authored
584 Statistics may be retrieved by accessing the
585 [Ostrich admin HTTP service](https://github.com/twitter/ostrich) on the admin HTTP port.
586 For example: `http://kestrel.host:2223/stats.json?period=60`.
587
588 Statistics are also available via the memcache protocol using the `STATS` command.
589
590
66c498b docs!
Robey Pointer authored
591 Kestrel as a library
592 --------------------
593
594 You can use kestrel as a library by just sticking the jar on your classpath.
595 It's a cheap way to get a durable work queue for inter-process or inter-thread
596 communication. Each queue is represented by a `PersistentQueue` object:
597
1531d0d finish cleaning up the guide.
Robey Pointer authored
598 class PersistentQueue(val name: String, persistencePath: String,
599 @volatile var config: QueueConfig, timer: Timer,
600 queueLookup: Option[(String => Option[PersistentQueue])]) {
66c498b docs!
Robey Pointer authored
601
602 and must be initialized before using:
603
604 def setup(): Unit
58746d2 more docs!
Robey Pointer authored
605
66c498b docs!
Robey Pointer authored
606 specifying the path for the journal files (if the queue will be journaled),
1531d0d finish cleaning up the guide.
Robey Pointer authored
607 the name of the queue, a `QueueConfig` object (derived from `QueueBuilder`),
608 a timer for handling timeout reads, and optionally a way to find other named
609 queues (for `expireToQueue` support).
58746d2 more docs!
Robey Pointer authored
610
66c498b docs!
Robey Pointer authored
611 To add an item to a queue:
58746d2 more docs!
Robey Pointer authored
612
1531d0d finish cleaning up the guide.
Robey Pointer authored
613 def add(value: Array[Byte], expiry: Option[Time]): Boolean
58746d2 more docs!
Robey Pointer authored
614
66c498b docs!
Robey Pointer authored
615 It will return `false` if the item was rejected because the queue was full.
0c1b88b lots more docs.
Robey Pointer authored
616
66c498b docs!
Robey Pointer authored
617 Queue items are represented by a case class:
618
1531d0d finish cleaning up the guide.
Robey Pointer authored
619 case class QItem(addTime: Time, expiry: Option[Time], data: Array[Byte], var xid: Int)
66c498b docs!
Robey Pointer authored
620
621 and several operations exist to remove or peek at the head item:
622
623 def peek(): Option[QItem]
624 def remove(): Option[QItem]
625
626 To open a reliable read, set `transaction` true, and later confirm or unremove
627 the item by its `xid`:
628
629 def remove(transaction: Boolean): Option[QItem]
630 def unremove(xid: Int)
631 def confirmRemove(xid: Int)
632
1531d0d finish cleaning up the guide.
Robey Pointer authored
633 You can also asynchronously remove or peek at items using futures.
66c498b docs!
Robey Pointer authored
634
1531d0d finish cleaning up the guide.
Robey Pointer authored
635 def waitRemove(deadline: Option[Time], transaction: Boolean): Future[Option[QItem]]
636 def waitPeek(deadline: Option[Time]): Future[Option[QItem]]
66c498b docs!
Robey Pointer authored
637
638 When done, you should close the queue:
639
640 def close(): Unit
641 def isClosed: Boolean
642
643 Here's a short example:
644
1531d0d finish cleaning up the guide.
Robey Pointer authored
645 var queue = new PersistentQueue("work", "/var/spool/kestrel", config, timer, None)
66c498b docs!
Robey Pointer authored
646 queue.setup()
647
648 // add an item with no expiration:
649 queue.add("hello".getBytes, 0)
650
651 // start to remove it, then back out:
652 val item = queue.remove(true)
653 queue.unremove(item.xid)
654
655 // remove an item with a 500msec timeout, and confirm it:
1531d0d finish cleaning up the guide.
Robey Pointer authored
656 queue.waitRemove(500.milliseconds.fromNow, true)() match {
657 case None =>
658 println("nothing. :(")
659 case Some(item) =>
660 println("got: " + new String(item.data))
661 queue.confirmRemove(item.xid)
66c498b docs!
Robey Pointer authored
662 }
0c1b88b lots more docs.
Robey Pointer authored
663
66c498b docs!
Robey Pointer authored
664 queue.close()
Something went wrong with that request. Please try again.