Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 567 lines (412 sloc) 23.172 kb
0c1b88b lots more docs.
Robey Pointer authored
1
62482e2 cleanup.
Robey Pointer authored
2 A working guide to kestrel
3 ==========================
0c1b88b lots more docs.
Robey Pointer authored
4
e224a6f Stephan Zuercher minor docs/guide.md update
zuercher authored
5 Kestrel is a very simple message queue that runs on the JVM. It supports multiple protocols:
6
7 - memcache: the memcache protocol, with some extensions
8 - thrift: Apache Thrift-based RPC
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
9 - text: a simple text-based protocol
0c1b88b lots more docs.
Robey Pointer authored
10
11 A single kestrel server has a set of queues identified by a name, which is
12 also the filename of that queue's journal file (usually in
13 `/var/spool/kestrel`). Each queue is a strictly-ordered FIFO of "items" of
14 binary data. Usually this data is in some serialized format like JSON or
15 ruby's marshal format.
16
d6199a9 explain queue names.
Robey Pointer authored
17 Generally queue names should be limited to alphanumerics `[A-Za-z0-9]`, dash
18 (`-`) and underline (`_`). In practice, kestrel doesn't enforce any
19 restrictions other than the name can't contain slash (`/`) because that can't
20 be used in filenames, squiggle (`~`) because it's used for temporary files,
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
21 plus (`+`) because it's used for fanout queues, and dot (`.`) because it's
22 reserved for future use. Queue names are case-sensitive, but if you're running
23 kestrel on OS X or Windows, you will want to refrain from taking advantage of
24 this, since the journal filenames on those two platforms are *not*
25 case-sensitive.
d6199a9 explain queue names.
Robey Pointer authored
26
0c1b88b lots more docs.
Robey Pointer authored
27 A cluster of kestrel servers is like a memcache cluster: the servers don't
28 know about each other, and don't do any cross-communication, so you can add as
29 many as you like. Clients have a list of all servers in the cluster, and pick
30 one at random for each operation. In this way, each queue appears to be spread
8c93d94 more doc improvements.
Robey Pointer authored
31 out across every server, with items in a loose ordering.
0c1b88b lots more docs.
Robey Pointer authored
32
58746d2 more docs!
Robey Pointer authored
33 When kestrel starts up, it scans the journal folder and creates queues based
34 on any journal files it finds there, to restore state to the way it was when
35 it last shutdown (or was killed or died). New queues are created by referring
36 to them (for example, adding or trying to remove an item). A queue can be
37 deleted with the "delete" command.
38
0c1b88b lots more docs.
Robey Pointer authored
39
40 Configuration
41 -------------
42
bf4ac4d Stephan Zuercher KEST-272: proper config inheritance; log queue/alias config on reload
zuercher authored
43 **NOTE:** Kestrel 2.3.4 introduces inheritance for queue configurations. For more
44 information, see below.
45
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
46 The config files for kestrel are scala expressions loaded at runtime, usually
47 from `production.scala`, although you can use `development.scala` by passing
48 `-Dstage=development` to the java command line.
49
50 The config file evaluates to a `KestrelConfig` object that's used to configure
51 the server as a whole, a default queue, and any overrides for specific named
52 queues. The fields on `KestrelConfig` are documented here with their default
53 values:
bf4ac4d Stephan Zuercher KEST-272: proper config inheritance; log queue/alias config on reload
zuercher authored
54 [KestrelConfig](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/KestrelConfig.html)
0c1b88b lots more docs.
Robey Pointer authored
55
56 To confirm the current configuration of each queue, send "dump_config" to
57 a server (which can be done over telnet).
58
59 To reload the config file on a running server, send "reload" the same way.
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
60 You should immediately see the changes in "dump_config", to confirm. Reloading
61 will only affect queue configuration, not global server configuration. To
62 change the server configuration, restart the server.
0c1b88b lots more docs.
Robey Pointer authored
63
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
64 Logging is configured according to `util-logging`. The logging configuration
65 syntax is described here:
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
66 [util-logging](https://github.com/twitter/util/blob/master/util-logging/README.markdown)
0c1b88b lots more docs.
Robey Pointer authored
67
bf4ac4d Stephan Zuercher KEST-272: proper config inheritance; log queue/alias config on reload
zuercher authored
68 Per-queue configuration options are documented here:
69 [QueueBuilder](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/QueueBuilder.html)
70
71 Queue alias configuration options are documented here:
72 [AliasBuilder](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/AliasBuilder.html)
73
74 Configuration Changes Starting in Kestrel 2.3.4
75 -----------------------------------------------
76
77 Starting with Kestrel 2.3.4, queue configurations are inherited:
78
79 * Any queue with no explict configuration (see `queues` in `KestrelConfig`) uses the default
80 queue configuration (see `default` in `KestrelConfig`). This behavior is unchanged from
81 previous versions.
9ffcd5c Stephan Zuercher minor guide edit
zuercher authored
82 * Any master (e.g. not fanout) queue with an explicit queue configuration overrides the default
83 queue configuration. For example, if `default.maxMemorySize` is set, all explicitly configured
bf4ac4d Stephan Zuercher KEST-272: proper config inheritance; log queue/alias config on reload
zuercher authored
84 queues will inherit that setting *unless* explicitly overridden in the queue's configuration.
85 Older versions of Kestrel *did not* apply values from the default queue configuration to any
86 explicitly configured queue.
87 * Any fanout queue (e.g., a queue with a `+` in its name), inherits its master queue's
88 configuration, unless explicitly overridden (see `queues` in `KestrelConfig`). Older versions
9ffcd5c Stephan Zuercher minor guide edit
zuercher authored
89 of Kestrel silently ignored explicit fanout queue configurations and used the master queue's
90 configuration.
bf4ac4d Stephan Zuercher KEST-272: proper config inheritance; log queue/alias config on reload
zuercher authored
91
92 ### Example Configuration
93 -------------------------
94
95 Existing configurations should continue to load, but the resulting configuration may
96 differ. As an example, the following configuration file and table illustrate the differences
97 between a configuration loaded by Kestrel 2.3.3 and Kestrel 2.3.4 (and later).
98
99 new KestrelConfig {
100 default.maxMemorySize = 8.megabytes
101
102 queues = new QueueBuilder() {
103 name = "q"
104 maxItems = 500
105 } :: new QueueBuilder() {
106 name = "q+fanout"
107 maxAge = 1.minute
108 } :: new QueueBuilder() {
109 name = "x"
110 maxMemorySize = 16.megabytes
111 }
112 }
113
114
115 <table>
fcabfac Stephan Zuercher fix angle brackets
zuercher authored
116 <tr><th>Queue</th> <th>Setting</th> <th>Kestrel &lt;= 2.3.3</th> <th>Kestrel &gt;= 2.3.4</th> </tr>
bf4ac4d Stephan Zuercher KEST-272: proper config inheritance; log queue/alias config on reload
zuercher authored
117 <tr><td>q</td> <td>maxMemorySize</td> <td>128.megabytes</td> <td>8.megabytes</td> </tr>
118 <tr><td>q+fanout</td> <td>maxMemorySize</td> <td>128.megabytes</td> <td>8.megabytes</td> </tr>
119 <tr><td>x</td> <td>maxMemorySize</td> <td>16.megabytes</td> <td>16.megabytes</td> </tr>
120 <tr><td>q</td> <td>maxItems</td> <td>500</td> <td>500</td> </tr>
121 <tr><td>q+fanout</td> <td>maxItems</td> <td>500</td> <td>500</td> </tr>
122 <tr><td>q+fanout</td> <td>maxAge</td> <td>None</td> <td>Some(1.minute)</td> </tr>
123 </table>
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
124
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
125
126 Full queues
127 -----------
128
129 A queue can have the following limits set on it:
130
131 - `maxItems` - total items in the queue
132 - `maxSize` - total bytes of data in the items in the queue
133
134 If either of these limits is reached, no new items can be added to the queue.
135 (Clients will receive an error when trying to add.) If you set
136 `discardOldWhenFull` to true, then all adds will succeed, and the oldest
137 item(s) will be silently discarded until the queue is back within the item
138 and size limits.
139
140 `maxItemSize` limits the size of any individual item. If an add is attempted
141 with an item larger than this limit, it always fails.
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
142
0c1b88b lots more docs.
Robey Pointer authored
143
144 The journal file
145 ----------------
146
147 The journal file is the only on-disk storage of a queue's contents, and it's
148 just a sequential record of each add or remove operation that's happened on
149 that queue. When kestrel starts up, it replays each queue's journal to build
150 up the in-memory queue that it uses for client queries.
151
58746d2 more docs!
Robey Pointer authored
152 The journal file is rotated in one of two conditions:
0c1b88b lots more docs.
Robey Pointer authored
153
1531d0d finish cleaning up the guide.
Robey Pointer authored
154 1. the queue is empty and the journal is larger than `defaultJournalSize`
0c1b88b lots more docs.
Robey Pointer authored
155
1531d0d finish cleaning up the guide.
Robey Pointer authored
156 2. the journal is larger than `maxJournalSize`
0c1b88b lots more docs.
Robey Pointer authored
157
1531d0d finish cleaning up the guide.
Robey Pointer authored
158 For example, if `defaultJournalSize` is 16MB (the default), then if the queue
159 is empty and the journal is larger than 16MB, it will be truncated into a new
160 (empty) file. If the journal is larger than `maxJournalSize` (1GB by default),
161 the journal will be rewritten periodically to contain just the live items.
0c1b88b lots more docs.
Robey Pointer authored
162
1531d0d finish cleaning up the guide.
Robey Pointer authored
163 You can turn the journal off for a queue (`keepJournal` = false) and the queue
0c1b88b lots more docs.
Robey Pointer authored
164 will exist only in memory. If the server restarts, all enqueued items are
1531d0d finish cleaning up the guide.
Robey Pointer authored
165 lost. You can also force a queue's journal to be sync'd to disk periodically,
166 or even after every write operation, at a performance cost, using
167 `syncJournal`.
0c1b88b lots more docs.
Robey Pointer authored
168
1531d0d finish cleaning up the guide.
Robey Pointer authored
169 If a queue grows past `maxMemorySize` bytes (128MB by default), only the
0c1b88b lots more docs.
Robey Pointer authored
170 first 128MB is kept in memory. The journal is used to track later items, and
171 as items are removed, the journal is played forward to keep 128MB in memory.
172 This is usually known as "read-behind" mode, but Twitter engineers sometimes
173 refer to it as the "square snake" because of the diagram used to brainstorm
174 the implementation. When a queue is in read-behind mode, removing an item will
175 often cause 2 disk operations instead of one: one to record the remove, and
176 one to read an item in from disk to keep 128MB in memory. This is the
177 trade-off to avoid filling memory and crashing the JVM.
178
179
58746d2 more docs!
Robey Pointer authored
180 Item expiration
181 ---------------
0c1b88b lots more docs.
Robey Pointer authored
182
58746d2 more docs!
Robey Pointer authored
183 When they come from a client, expiration times are handled in the same way as
184 memcache: if the number is small (less than one million), it's interpreted as
185 a relative number of seconds from now. Otherwise it's interpreted as an
186 absolute unix epoch time, in seconds since the beginning of 1 January 1970
187 GMT.
0c1b88b lots more docs.
Robey Pointer authored
188
58746d2 more docs!
Robey Pointer authored
189 Expiration times are immediately translated into an absolute time, in
1531d0d finish cleaning up the guide.
Robey Pointer authored
190 *milliseconds*, and if it's further in the future than the queue's `maxAge`,
a3c112f fix typos
Robey Pointer authored
191 the `maxAge` is used instead. An expiration of 0, which is usually the
58746d2 more docs!
Robey Pointer authored
192 default, means an item never expires.
0c1b88b lots more docs.
Robey Pointer authored
193
58746d2 more docs!
Robey Pointer authored
194 Expired items are flushed from a queue whenever a new item is added or
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
195 removed. Additionally, if the global config option `expirationTimerFrequency`
196 is set, a background thread will periodically remove expired items from the
197 head of each queue. The provided `production.conf` sets this to one second.
198 If this is turned off, an idle queue won't have any items expired, but you
199 can still trigger a check by doing a "peek" on it.
200
201 Normally, expired items are discarded. If `expireToQueue` is set, then
202 expired items are moved to the specified queue just as if a client had put
203 it there. The item is added with no expiration time, but that can be
204 overridden if the new queue has a default expiration policy.
205
206 To prevent stalling the server when it encounters a swarm of items that all
207 expired at the same time, `maxExpireSweep` limits the number of items that
208 will be removed by the background thread in a single round. This is primarily
209 useful as a throttling mechanism when using a queue as a way to delay work.
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
210
7841145 Cory G Watson Expiring queues.
gphat authored
211 Queue expiration
212 ----------------
213
e224a6f Stephan Zuercher minor docs/guide.md update
zuercher authored
214 Whole queues can be configured to expire as well. If `maxQueueAge` is set
52c5fac Cory G Watson Modify various bit after review by robey.
gphat authored
215 `expirationTimerFrequency` is used to check the queue age. If the queue is
216 empty, and it has been longer than `maxQueueAge` since it was created then
217 the queue will be deleted.
0c1b88b lots more docs.
Robey Pointer authored
218
e224a6f Stephan Zuercher minor docs/guide.md update
zuercher authored
219
58746d2 more docs!
Robey Pointer authored
220 Fanout Queues
221 -------------
222
223 If a queue name has a `+` in it (like "`orders+audit`"), it's treated as a
224 fanout queue, using the format `<parent>+<child>`. These queues belong to a
225 parent queue -- in this example, the "orders" queue. Every item written into
226 a parent queue will also be written into each of its children.
227
228 Fanout queues each have their own journal file (if the parent queue has a
229 journal file) and otherwise behave exactly like any other queue. You can get
230 and peek and even add items directly to a child queue if you want. It uses the
231 parent queue's configuration instead of having independent child queue
232 configuration blocks.
233
234 When a fanout queue is first referenced by a client, the journal file (if any)
235 is created, and it will start receiving new items written to the parent queue.
236 Existing items are not copied over. A fanout queue can be deleted to stop it
237 from receiving new items.
238
6993502 Document fanoutOnly.
Cory Watson authored
239 `fanoutOnly` may be set to true if the queue in question will only serve write
240 point for fanout queues. No journal file will be kept for the parent, only
241 for the child queues. This saves the overhead of writing to the parent and
242 removes the need to empty it. Note that setting `fanoutOnly` to true and
243 having no fanouts for the queue effectively makes it a black hole.
e224a6f Stephan Zuercher minor docs/guide.md update
zuercher authored
244
ff976b0 Stephan Zuercher introduce queue aliases (including stats)
zuercher authored
245 Queue Aliases
246 -------------
247
248 Queue aliases are somewhat similar to fanout queues, but without a required
249 naming convention or implicit creation of child queues. A queue alias can
250 only be used in set operations. Kestrel responds to attempts to retrieve
251 items from the alias as if it were an empty queue. Delete and flush requests
252 are also ignored.
58746d2 more docs!
Robey Pointer authored
253
e224a6f Stephan Zuercher minor docs/guide.md update
zuercher authored
254
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
255 Protocols
256 ---------
257
258 Kestrel supports three protocols: memcache, thrift and text. The
259 [Finagle project](http://twitter.github.com/finagle/) can be used to connect clients
260 to a Kestrel server via the memcache or thrift protocols.
bb7eb5e rename xid to id in the thrift interface, and link it from the docs
Robey Pointer authored
261
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
262 ### Thrift
263 ----------
264
265 The thrift protocol is documented in the thrift IDL:
bb7eb5e rename xid to id in the thrift interface, and link it from the docs
Robey Pointer authored
266 [kestrel.thrift](https://github.com/robey/kestrel/blob/master/src/main/thrift/kestrel.thrift)
267
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
268 Reliable reads via the thrift protocol are specified by indicating how long the server
269 should wait before aborting the unacknowledged read.
bb7eb5e rename xid to id in the thrift interface, and link it from the docs
Robey Pointer authored
270
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
271
272 ### Memcache
273 ------------
274
275 The official memcache protocol is described here:
276 [protocol.txt](https://github.com/memcached/memcached/blob/master/doc/protocol.txt)
277
278 The kestrel implementation of the memcache protocol commands is described below.
58746d2 more docs!
Robey Pointer authored
279
280 - `SET <queue-name> <flags (ignored)> <expiration> <# bytes>`
281
282 Add an item to a queue. It may fail if the queue has a size or item limit
283 and it's full.
284
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
285 - `GET <queue-name>[options]`
66c498b docs!
Robey Pointer authored
286
58746d2 more docs!
Robey Pointer authored
287 Remove an item from a queue. It will return an empty response immediately if
288 the queue is empty. The queue name may be followed by options separated
289 by `/`:
290
66c498b docs!
Robey Pointer authored
291 - `/t=<milliseconds>`
292
293 Wait up to a given time limit for a new item to arrive. If an item arrives
294 on the queue within this timeout, it's returned as normal. Otherwise,
295 after that timeout, an empty response is returned.
296
297 - `/open`
298
299 Tentatively remove an item from the queue. The item is returned as usual
300 but is also set aside in case the client disappears before sending a
301 "close" request. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
302
66c498b docs!
Robey Pointer authored
303 - `/close`
58746d2 more docs!
Robey Pointer authored
304
66c498b docs!
Robey Pointer authored
305 Close any existing open read. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
306
66c498b docs!
Robey Pointer authored
307 - `/abort`
58746d2 more docs!
Robey Pointer authored
308
66c498b docs!
Robey Pointer authored
309 Cancel any existing open read, returing that item to the head of the
310 queue. It will be the next item fetched. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
311
66c498b docs!
Robey Pointer authored
312 - `/peek`
58746d2 more docs!
Robey Pointer authored
313
66c498b docs!
Robey Pointer authored
314 Return the first available item from the queue, if there is one, but don't
315 remove it. You can't combine this with any of the reliable read options.
58746d2 more docs!
Robey Pointer authored
316
d1b4ee7 R. Tyler Croy Update guide.md
rtyler authored
317 For example, to open a new read, waiting up to 500msec for an item:
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
318
319 GET work/t=500/open
320
321 Or to close an existing read and open a new one:
322
323 GET work/close/open
324
58746d2 more docs!
Robey Pointer authored
325 - `DELETE <queue-name>`
326
327 Drop a queue, discarding any items in it, and deleting any associated
328 journal files.
329
330 - `FLUSH <queue-name>`
331
332 Discard all items remaining in this queue. The queue remains live and new
333 items can be added. The time it takes to flush will be linear to the current
334 queue size, and any other activity on this queue will block while it's being
335 flushed.
336
337 - `FLUSH_ALL`
338
339 Discard all items remaining in all queues. The queues are flushed one at a
340 time, as if kestrel received a `FLUSH` command for each queue.
0c1b88b lots more docs.
Robey Pointer authored
341
58746d2 more docs!
Robey Pointer authored
342 - `VERSION`
343
344 Display the kestrel version in a way compatible with memcache.
345
346 - `SHUTDOWN`
347
348 Cleanly shutdown the server and exit.
349
350 - `RELOAD`
351
352 Reload the config file and reconfigure all queues. This should have no
353 noticable effect on the server's responsiveness.
354
355 - `STATS`
356
357 Display server stats in memcache style. They're described below.
358
359 - `DUMP_STATS`
360
361 Display server stats in a more readable style, grouped by queue. They're
362 described below.
363
55c7342 allow memcache protocol to have an optional max_items on "monitor".
Robey Pointer authored
364 - `MONITOR <queue-name> <seconds> [max-items]`
1531d0d finish cleaning up the guide.
Robey Pointer authored
365
e224a6f Stephan Zuercher minor docs/guide.md update
zuercher authored
366 Monitor a queue for a time, fetching any new items that arrive, up to an
367 optional maximum number of items. Clients are queued in a fair fashion,
368 per-item, so many clients may monitor a queue at once. After the given
369 timeout, a separate `END` response will signal the end of the monitor
370 period. Any fetched items are open transactions (see "Reliable Reads"
371 below), and should be closed with `CONFIRM`.
1531d0d finish cleaning up the guide.
Robey Pointer authored
372
373 - `CONFIRM <queue-name> <count>`
374
375 Confirm receipt of `count` items from a queue. Usually this is the response
376 to a `MONITOR` command, to confirm the items that arrived during the monitor
377 period.
378
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
379
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
380 #### Reliable reads
381 -------------------
58746d2 more docs!
Robey Pointer authored
382
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
383 Note: this section is specific to the memcache protocol.
e224a6f Stephan Zuercher minor docs/guide.md update
zuercher authored
384
58746d2 more docs!
Robey Pointer authored
385 Normally when a client removes an item from the queue, kestrel immediately
386 discards the item and assumes the client has taken ownership. This isn't
387 always safe, because a client could crash or lose the network connection
388 before it gets the item. So kestrel also supports a "reliable read" that
389 happens in two stages, using the `/open` and `/close` options to `GET`.
390
391 When `/open` is used, and an item is available, kestrel will remove it from
392 the queue and send it to the client as usual. But it will also set the item
393 aside. If a client disconnects while it has an open read, the item is put back
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
394 into the queue, at the head, so it will be the next item fetched. Only one
395 item can be "open" per client connection.
58746d2 more docs!
Robey Pointer authored
396
397 A previous open request is closed with `/close`. The server will reject any
398 attempt to open another read when one is already open, but it will ignore
399 `/close` if there's no open request, so that you can add `/close` to every
400 `GET` request for convenience.
401
402 If for some reason you want to abort a read without disconnecting, you can use
403 `/abort`. But because aborted items are placed back at the head of the queue,
404 this isn't a good way to deal with client errors. Since the error-causing item
405 will always be the next one available, you'll end up bouncing the same item
406 around between clients instead of making progress.
407
408 There's always a trade-off: either potentially lose items or potentially
409 receive the same item multiple times. Reliable reads choose the latter option.
410 To use this tactic successfully, work items should be idempotent, meaning the
411 work could be done 2 or 3 times and have the same effect as if it had been
412 done only once (except wasting some resources).
413
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
414 Example:
415
416 GET dirty_jobs/close/open
417 (receives job 1)
418 GET dirty_jobs/close/open
419 (closes job 1, receives job 2)
420 ...etc...
58746d2 more docs!
Robey Pointer authored
421
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
422
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
423 ### Text protocol
424 -----------------
425
426 Kestrel supports a limited, text-only protocol. You are encouraged to use the
427 memcache protocol instead.
428
429 The text protocol does not support reliable reads.
430
431
58746d2 more docs!
Robey Pointer authored
432 Server stats
433 ------------
434
66c498b docs!
Robey Pointer authored
435 Global stats reported by kestrel are:
436
437 - `uptime` - seconds the server has been online
438 - `time` - current time in unix epoch
439 - `version` - version string, like "1.2"
440 - `curr_items` - total of items waiting in all queues
441 - `total_itmes` - total of items that have ever been added in this server's
442 lifetime
443 - `bytes` - total byte size of items waiting in all queues
444 - `curr_connections` - current open connections from clients
445 - `total_connections` - total connections that have been opened in this
446 server's lifetime
447 - `cmd_get` - total `GET` requests
448 - `cmd_set` - total `SET` requests
449 - `cmd_peek` - total `GET/peek` requests
450 - `get_hits` - total `GET` requests that received an item
451 - `get_misses` - total `GET` requests on an empty queue
452 - `bytes_read` - total bytes read from clients
453 - `bytes_written` - total bytes written to clients
0d8a507 Cory G Watson Add more statistics along with tests and docs.
gphat authored
454 - `queue_creates` - total number of queues created
7841145 Cory G Watson Expiring queues.
gphat authored
455 - `queue_deletes` - total number of queues deleted (includes expires)
456 - `queue_expires` - total number of queues expires
66c498b docs!
Robey Pointer authored
457
458 For each queue, the following stats are also reported:
459
460 - `items` - items waiting in this queue
461 - `bytes` - total byte size of items waiting in this queue
462 - `total_items` - total items that have been added to this queue in this
463 server's lifetime
464 - `logsize` - byte size of the queue's journal file
465 - `expired_items` - total items that have been expired from this queue in this
466 server's lifetime
467 - `mem_items` - items in this queue that are currently in memory
468 - `mem_bytes` - total byte size of items in this queue that are currently in
469 memory (will always be less than or equal to `max_memory_size` config for
470 the queue)
471 - `age` - time, in milliseconds, that the last item to be fetched from this
472 queue had been waiting; that is, the time between `SET` and `GET`; if the
473 queue is empty, this will always be zero
474 - `discarded` - number of items discarded because the queue was too full
475 - `waiters` - number of clients waiting for an item from this queue (using
476 `GET/t`)
721bf6b document the new stat.
Robey Pointer authored
477 - `open_transactions` - items read with `/open` but not yet confirmed
9afe5c4 Stephan Zuercher track number of transactional get requests and cancellations per queue
zuercher authored
478 - `transactions` - number of transactional get requests (irrespective of whether an
479 item was read or not)
480 - `canceled_transactions` - number of transactional get requests canceled (for any
481 reason)
5e4a681 Cory G Watson Add create_time as a queue stat and document age_msec.
gphat authored
482 - `total_flushes` - total number of times this queue has been flushed
483 - `age_msec` - age of the last item read from the queue
484 - `create_time` - the time that the queue was created (in milliseconds since epoch)
66c498b docs!
Robey Pointer authored
485
c66bbb3 Stephan Zuercher more docs/guide.md cleanup
zuercher authored
486 Statistics may be retrieved by accessing the
487 [Ostrich admin HTTP service](https://github.com/twitter/ostrich) on the admin HTTP port.
488 For example: `http://kestrel.host:2223/stats.json?period=60`.
489
490 Statistics are also available via the memcache protocol using the `STATS` command.
491
492
66c498b docs!
Robey Pointer authored
493 Kestrel as a library
494 --------------------
495
496 You can use kestrel as a library by just sticking the jar on your classpath.
497 It's a cheap way to get a durable work queue for inter-process or inter-thread
498 communication. Each queue is represented by a `PersistentQueue` object:
499
1531d0d finish cleaning up the guide.
Robey Pointer authored
500 class PersistentQueue(val name: String, persistencePath: String,
501 @volatile var config: QueueConfig, timer: Timer,
502 queueLookup: Option[(String => Option[PersistentQueue])]) {
66c498b docs!
Robey Pointer authored
503
504 and must be initialized before using:
505
506 def setup(): Unit
58746d2 more docs!
Robey Pointer authored
507
66c498b docs!
Robey Pointer authored
508 specifying the path for the journal files (if the queue will be journaled),
1531d0d finish cleaning up the guide.
Robey Pointer authored
509 the name of the queue, a `QueueConfig` object (derived from `QueueBuilder`),
510 a timer for handling timeout reads, and optionally a way to find other named
511 queues (for `expireToQueue` support).
58746d2 more docs!
Robey Pointer authored
512
66c498b docs!
Robey Pointer authored
513 To add an item to a queue:
58746d2 more docs!
Robey Pointer authored
514
1531d0d finish cleaning up the guide.
Robey Pointer authored
515 def add(value: Array[Byte], expiry: Option[Time]): Boolean
58746d2 more docs!
Robey Pointer authored
516
66c498b docs!
Robey Pointer authored
517 It will return `false` if the item was rejected because the queue was full.
0c1b88b lots more docs.
Robey Pointer authored
518
66c498b docs!
Robey Pointer authored
519 Queue items are represented by a case class:
520
1531d0d finish cleaning up the guide.
Robey Pointer authored
521 case class QItem(addTime: Time, expiry: Option[Time], data: Array[Byte], var xid: Int)
66c498b docs!
Robey Pointer authored
522
523 and several operations exist to remove or peek at the head item:
524
525 def peek(): Option[QItem]
526 def remove(): Option[QItem]
527
528 To open a reliable read, set `transaction` true, and later confirm or unremove
529 the item by its `xid`:
530
531 def remove(transaction: Boolean): Option[QItem]
532 def unremove(xid: Int)
533 def confirmRemove(xid: Int)
534
1531d0d finish cleaning up the guide.
Robey Pointer authored
535 You can also asynchronously remove or peek at items using futures.
66c498b docs!
Robey Pointer authored
536
1531d0d finish cleaning up the guide.
Robey Pointer authored
537 def waitRemove(deadline: Option[Time], transaction: Boolean): Future[Option[QItem]]
538 def waitPeek(deadline: Option[Time]): Future[Option[QItem]]
66c498b docs!
Robey Pointer authored
539
540 When done, you should close the queue:
541
542 def close(): Unit
543 def isClosed: Boolean
544
545 Here's a short example:
546
1531d0d finish cleaning up the guide.
Robey Pointer authored
547 var queue = new PersistentQueue("work", "/var/spool/kestrel", config, timer, None)
66c498b docs!
Robey Pointer authored
548 queue.setup()
549
550 // add an item with no expiration:
551 queue.add("hello".getBytes, 0)
552
553 // start to remove it, then back out:
554 val item = queue.remove(true)
555 queue.unremove(item.xid)
556
557 // remove an item with a 500msec timeout, and confirm it:
1531d0d finish cleaning up the guide.
Robey Pointer authored
558 queue.waitRemove(500.milliseconds.fromNow, true)() match {
559 case None =>
560 println("nothing. :(")
561 case Some(item) =>
562 println("got: " + new String(item.data))
563 queue.confirmRemove(item.xid)
66c498b docs!
Robey Pointer authored
564 }
0c1b88b lots more docs.
Robey Pointer authored
565
66c498b docs!
Robey Pointer authored
566 queue.close()
Something went wrong with that request. Please try again.