Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 471 lines (339 sloc) 18.854 kb
0c1b88b lots more docs.
Robey Pointer authored
1
62482e2 cleanup.
Robey Pointer authored
2 A working guide to kestrel
3 ==========================
0c1b88b lots more docs.
Robey Pointer authored
4
e224a6f @zuercher minor docs/guide.md update
zuercher authored
5 Kestrel is a very simple message queue that runs on the JVM. It supports multiple protocols:
6
7 - memcache: the memcache protocol, with some extensions
8 - thrift: Apache Thrift-based RPC
0c1b88b lots more docs.
Robey Pointer authored
9
10 A single kestrel server has a set of queues identified by a name, which is
11 also the filename of that queue's journal file (usually in
12 `/var/spool/kestrel`). Each queue is a strictly-ordered FIFO of "items" of
13 binary data. Usually this data is in some serialized format like JSON or
14 ruby's marshal format.
15
d6199a9 explain queue names.
Robey Pointer authored
16 Generally queue names should be limited to alphanumerics `[A-Za-z0-9]`, dash
17 (`-`) and underline (`_`). In practice, kestrel doesn't enforce any
18 restrictions other than the name can't contain slash (`/`) because that can't
19 be used in filenames, squiggle (`~`) because it's used for temporary files,
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
20 plus (`+`) because it's used for fanout queues, and dot (`.`) because it's
21 reserved for future use. Queue names are case-sensitive, but if you're running
22 kestrel on OS X or Windows, you will want to refrain from taking advantage of
23 this, since the journal filenames on those two platforms are *not*
24 case-sensitive.
d6199a9 explain queue names.
Robey Pointer authored
25
0c1b88b lots more docs.
Robey Pointer authored
26 A cluster of kestrel servers is like a memcache cluster: the servers don't
27 know about each other, and don't do any cross-communication, so you can add as
28 many as you like. Clients have a list of all servers in the cluster, and pick
29 one at random for each operation. In this way, each queue appears to be spread
8c93d94 more doc improvements.
Robey Pointer authored
30 out across every server, with items in a loose ordering.
0c1b88b lots more docs.
Robey Pointer authored
31
58746d2 more docs!
Robey Pointer authored
32 When kestrel starts up, it scans the journal folder and creates queues based
33 on any journal files it finds there, to restore state to the way it was when
34 it last shutdown (or was killed or died). New queues are created by referring
35 to them (for example, adding or trying to remove an item). A queue can be
36 deleted with the "delete" command.
37
0c1b88b lots more docs.
Robey Pointer authored
38
39 Configuration
40 -------------
41
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
42 The config files for kestrel are scala expressions loaded at runtime, usually
43 from `production.scala`, although you can use `development.scala` by passing
44 `-Dstage=development` to the java command line.
45
46 The config file evaluates to a `KestrelConfig` object that's used to configure
47 the server as a whole, a default queue, and any overrides for specific named
48 queues. The fields on `KestrelConfig` are documented here with their default
49 values:
e224a6f @zuercher minor docs/guide.md update
zuercher authored
50 [KestrelConfig.html](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/KestrelConfig.html)
0c1b88b lots more docs.
Robey Pointer authored
51
52 To confirm the current configuration of each queue, send "dump_config" to
53 a server (which can be done over telnet).
54
55 To reload the config file on a running server, send "reload" the same way.
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
56 You should immediately see the changes in "dump_config", to confirm. Reloading
57 will only affect queue configuration, not global server configuration. To
58 change the server configuration, restart the server.
0c1b88b lots more docs.
Robey Pointer authored
59
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
60 Logging is configured according to `util-logging`. The logging configuration
61 syntax is described here:
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
62 [util-logging](https://github.com/twitter/util/blob/master/util-logging/README.markdown)
0c1b88b lots more docs.
Robey Pointer authored
63
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
64 Per-queue configuration is documented here:
e224a6f @zuercher minor docs/guide.md update
zuercher authored
65 [QueueBuilder.html](http://robey.github.com/kestrel/api/main/api/net/lag/kestrel/config/QueueBuilder.html)
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
66
67
68 Full queues
69 -----------
70
71 A queue can have the following limits set on it:
72
73 - `maxItems` - total items in the queue
74 - `maxSize` - total bytes of data in the items in the queue
75
76 If either of these limits is reached, no new items can be added to the queue.
77 (Clients will receive an error when trying to add.) If you set
78 `discardOldWhenFull` to true, then all adds will succeed, and the oldest
79 item(s) will be silently discarded until the queue is back within the item
80 and size limits.
81
82 `maxItemSize` limits the size of any individual item. If an add is attempted
83 with an item larger than this limit, it always fails.
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
84
0c1b88b lots more docs.
Robey Pointer authored
85
86 The journal file
87 ----------------
88
89 The journal file is the only on-disk storage of a queue's contents, and it's
90 just a sequential record of each add or remove operation that's happened on
91 that queue. When kestrel starts up, it replays each queue's journal to build
92 up the in-memory queue that it uses for client queries.
93
58746d2 more docs!
Robey Pointer authored
94 The journal file is rotated in one of two conditions:
0c1b88b lots more docs.
Robey Pointer authored
95
1531d0d finish cleaning up the guide.
Robey Pointer authored
96 1. the queue is empty and the journal is larger than `defaultJournalSize`
0c1b88b lots more docs.
Robey Pointer authored
97
1531d0d finish cleaning up the guide.
Robey Pointer authored
98 2. the journal is larger than `maxJournalSize`
0c1b88b lots more docs.
Robey Pointer authored
99
1531d0d finish cleaning up the guide.
Robey Pointer authored
100 For example, if `defaultJournalSize` is 16MB (the default), then if the queue
101 is empty and the journal is larger than 16MB, it will be truncated into a new
102 (empty) file. If the journal is larger than `maxJournalSize` (1GB by default),
103 the journal will be rewritten periodically to contain just the live items.
0c1b88b lots more docs.
Robey Pointer authored
104
1531d0d finish cleaning up the guide.
Robey Pointer authored
105 You can turn the journal off for a queue (`keepJournal` = false) and the queue
0c1b88b lots more docs.
Robey Pointer authored
106 will exist only in memory. If the server restarts, all enqueued items are
1531d0d finish cleaning up the guide.
Robey Pointer authored
107 lost. You can also force a queue's journal to be sync'd to disk periodically,
108 or even after every write operation, at a performance cost, using
109 `syncJournal`.
0c1b88b lots more docs.
Robey Pointer authored
110
1531d0d finish cleaning up the guide.
Robey Pointer authored
111 If a queue grows past `maxMemorySize` bytes (128MB by default), only the
0c1b88b lots more docs.
Robey Pointer authored
112 first 128MB is kept in memory. The journal is used to track later items, and
113 as items are removed, the journal is played forward to keep 128MB in memory.
114 This is usually known as "read-behind" mode, but Twitter engineers sometimes
115 refer to it as the "square snake" because of the diagram used to brainstorm
116 the implementation. When a queue is in read-behind mode, removing an item will
117 often cause 2 disk operations instead of one: one to record the remove, and
118 one to read an item in from disk to keep 128MB in memory. This is the
119 trade-off to avoid filling memory and crashing the JVM.
120
121
58746d2 more docs!
Robey Pointer authored
122 Item expiration
123 ---------------
0c1b88b lots more docs.
Robey Pointer authored
124
58746d2 more docs!
Robey Pointer authored
125 When they come from a client, expiration times are handled in the same way as
126 memcache: if the number is small (less than one million), it's interpreted as
127 a relative number of seconds from now. Otherwise it's interpreted as an
128 absolute unix epoch time, in seconds since the beginning of 1 January 1970
129 GMT.
0c1b88b lots more docs.
Robey Pointer authored
130
58746d2 more docs!
Robey Pointer authored
131 Expiration times are immediately translated into an absolute time, in
1531d0d finish cleaning up the guide.
Robey Pointer authored
132 *milliseconds*, and if it's further in the future than the queue's `maxAge`,
a3c112f fix typos
Robey Pointer authored
133 the `maxAge` is used instead. An expiration of 0, which is usually the
58746d2 more docs!
Robey Pointer authored
134 default, means an item never expires.
0c1b88b lots more docs.
Robey Pointer authored
135
58746d2 more docs!
Robey Pointer authored
136 Expired items are flushed from a queue whenever a new item is added or
fd3f9d4 add more docs and clean them up a bit
Robey Pointer authored
137 removed. Additionally, if the global config option `expirationTimerFrequency`
138 is set, a background thread will periodically remove expired items from the
139 head of each queue. The provided `production.conf` sets this to one second.
140 If this is turned off, an idle queue won't have any items expired, but you
141 can still trigger a check by doing a "peek" on it.
142
143 Normally, expired items are discarded. If `expireToQueue` is set, then
144 expired items are moved to the specified queue just as if a client had put
145 it there. The item is added with no expiration time, but that can be
146 overridden if the new queue has a default expiration policy.
147
148 To prevent stalling the server when it encounters a swarm of items that all
149 expired at the same time, `maxExpireSweep` limits the number of items that
150 will be removed by the background thread in a single round. This is primarily
151 useful as a throttling mechanism when using a queue as a way to delay work.
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
152
7841145 @gphat Expiring queues.
gphat authored
153 Queue expiration
154 ----------------
155
e224a6f @zuercher minor docs/guide.md update
zuercher authored
156 Whole queues can be configured to expire as well. If `maxQueueAge` is set
52c5fac @gphat Modify various bit after review by robey.
gphat authored
157 `expirationTimerFrequency` is used to check the queue age. If the queue is
158 empty, and it has been longer than `maxQueueAge` since it was created then
159 the queue will be deleted.
0c1b88b lots more docs.
Robey Pointer authored
160
e224a6f @zuercher minor docs/guide.md update
zuercher authored
161
58746d2 more docs!
Robey Pointer authored
162 Fanout Queues
163 -------------
164
165 If a queue name has a `+` in it (like "`orders+audit`"), it's treated as a
166 fanout queue, using the format `<parent>+<child>`. These queues belong to a
167 parent queue -- in this example, the "orders" queue. Every item written into
168 a parent queue will also be written into each of its children.
169
170 Fanout queues each have their own journal file (if the parent queue has a
171 journal file) and otherwise behave exactly like any other queue. You can get
172 and peek and even add items directly to a child queue if you want. It uses the
173 parent queue's configuration instead of having independent child queue
174 configuration blocks.
175
176 When a fanout queue is first referenced by a client, the journal file (if any)
177 is created, and it will start receiving new items written to the parent queue.
178 Existing items are not copied over. A fanout queue can be deleted to stop it
179 from receiving new items.
180
e224a6f @zuercher minor docs/guide.md update
zuercher authored
181
ff976b0 @zuercher introduce queue aliases (including stats)
zuercher authored
182 Queue Aliases
183 -------------
184
185 Queue aliases are somewhat similar to fanout queues, but without a required
186 naming convention or implicit creation of child queues. A queue alias can
187 only be used in set operations. Kestrel responds to attempts to retrieve
188 items from the alias as if it were an empty queue. Delete and flush requests
189 are also ignored.
58746d2 more docs!
Robey Pointer authored
190
e224a6f @zuercher minor docs/guide.md update
zuercher authored
191
bb7eb5e rename xid to id in the thrift interface, and link it from the docs
Robey Pointer authored
192 Thrift protocol
193 ---------------
194
195 The thrift protocol is documented in the thrift file here:
196 [kestrel.thrift](https://github.com/robey/kestrel/blob/master/src/main/thrift/kestrel.thrift)
197
198
58746d2 more docs!
Robey Pointer authored
199 Memcache commands
200 -----------------
201
202 - `SET <queue-name> <flags (ignored)> <expiration> <# bytes>`
203
204 Add an item to a queue. It may fail if the queue has a size or item limit
205 and it's full.
206
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
207 - `GET <queue-name>[options]`
66c498b docs!
Robey Pointer authored
208
58746d2 more docs!
Robey Pointer authored
209 Remove an item from a queue. It will return an empty response immediately if
210 the queue is empty. The queue name may be followed by options separated
211 by `/`:
212
66c498b docs!
Robey Pointer authored
213 - `/t=<milliseconds>`
214
215 Wait up to a given time limit for a new item to arrive. If an item arrives
216 on the queue within this timeout, it's returned as normal. Otherwise,
217 after that timeout, an empty response is returned.
218
219 - `/open`
220
221 Tentatively remove an item from the queue. The item is returned as usual
222 but is also set aside in case the client disappears before sending a
223 "close" request. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
224
66c498b docs!
Robey Pointer authored
225 - `/close`
58746d2 more docs!
Robey Pointer authored
226
66c498b docs!
Robey Pointer authored
227 Close any existing open read. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
228
66c498b docs!
Robey Pointer authored
229 - `/abort`
58746d2 more docs!
Robey Pointer authored
230
66c498b docs!
Robey Pointer authored
231 Cancel any existing open read, returing that item to the head of the
232 queue. It will be the next item fetched. (See "Reliable Reads" below.)
58746d2 more docs!
Robey Pointer authored
233
66c498b docs!
Robey Pointer authored
234 - `/peek`
58746d2 more docs!
Robey Pointer authored
235
66c498b docs!
Robey Pointer authored
236 Return the first available item from the queue, if there is one, but don't
237 remove it. You can't combine this with any of the reliable read options.
58746d2 more docs!
Robey Pointer authored
238
d1b4ee7 @rtyler Update guide.md
rtyler authored
239 For example, to open a new read, waiting up to 500msec for an item:
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
240
241 GET work/t=500/open
242
243 Or to close an existing read and open a new one:
244
245 GET work/close/open
246
58746d2 more docs!
Robey Pointer authored
247 - `DELETE <queue-name>`
248
249 Drop a queue, discarding any items in it, and deleting any associated
250 journal files.
251
252 - `FLUSH <queue-name>`
253
254 Discard all items remaining in this queue. The queue remains live and new
255 items can be added. The time it takes to flush will be linear to the current
256 queue size, and any other activity on this queue will block while it's being
257 flushed.
258
259 - `FLUSH_ALL`
260
261 Discard all items remaining in all queues. The queues are flushed one at a
262 time, as if kestrel received a `FLUSH` command for each queue.
0c1b88b lots more docs.
Robey Pointer authored
263
58746d2 more docs!
Robey Pointer authored
264 - `VERSION`
265
266 Display the kestrel version in a way compatible with memcache.
267
268 - `SHUTDOWN`
269
270 Cleanly shutdown the server and exit.
271
272 - `RELOAD`
273
274 Reload the config file and reconfigure all queues. This should have no
275 noticable effect on the server's responsiveness.
276
277 - `STATS`
278
279 Display server stats in memcache style. They're described below.
280
281 - `DUMP_STATS`
282
283 Display server stats in a more readable style, grouped by queue. They're
284 described below.
285
55c7342 allow memcache protocol to have an optional max_items on "monitor".
Robey Pointer authored
286 - `MONITOR <queue-name> <seconds> [max-items]`
1531d0d finish cleaning up the guide.
Robey Pointer authored
287
e224a6f @zuercher minor docs/guide.md update
zuercher authored
288 Monitor a queue for a time, fetching any new items that arrive, up to an
289 optional maximum number of items. Clients are queued in a fair fashion,
290 per-item, so many clients may monitor a queue at once. After the given
291 timeout, a separate `END` response will signal the end of the monitor
292 period. Any fetched items are open transactions (see "Reliable Reads"
293 below), and should be closed with `CONFIRM`.
1531d0d finish cleaning up the guide.
Robey Pointer authored
294
295 - `CONFIRM <queue-name> <count>`
296
297 Confirm receipt of `count` items from a queue. Usually this is the response
298 to a `MONITOR` command, to confirm the items that arrived during the monitor
299 period.
300
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
301
58746d2 more docs!
Robey Pointer authored
302 Reliable reads
303 --------------
304
e224a6f @zuercher minor docs/guide.md update
zuercher authored
305 This section describes the implementation of reliable reads for the memcache
306 protocol. See the thrift protocol documentation for details on using reliable
307 reads with that protocol.
308
58746d2 more docs!
Robey Pointer authored
309 Normally when a client removes an item from the queue, kestrel immediately
310 discards the item and assumes the client has taken ownership. This isn't
311 always safe, because a client could crash or lose the network connection
312 before it gets the item. So kestrel also supports a "reliable read" that
313 happens in two stages, using the `/open` and `/close` options to `GET`.
314
315 When `/open` is used, and an item is available, kestrel will remove it from
316 the queue and send it to the client as usual. But it will also set the item
317 aside. If a client disconnects while it has an open read, the item is put back
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
318 into the queue, at the head, so it will be the next item fetched. Only one
319 item can be "open" per client connection.
58746d2 more docs!
Robey Pointer authored
320
321 A previous open request is closed with `/close`. The server will reject any
322 attempt to open another read when one is already open, but it will ignore
323 `/close` if there's no open request, so that you can add `/close` to every
324 `GET` request for convenience.
325
326 If for some reason you want to abort a read without disconnecting, you can use
327 `/abort`. But because aborted items are placed back at the head of the queue,
328 this isn't a good way to deal with client errors. Since the error-causing item
329 will always be the next one available, you'll end up bouncing the same item
330 around between clients instead of making progress.
331
332 There's always a trade-off: either potentially lose items or potentially
333 receive the same item multiple times. Reliable reads choose the latter option.
334 To use this tactic successfully, work items should be idempotent, meaning the
335 work could be done 2 or 3 times and have the same effect as if it had been
336 done only once (except wasting some resources).
337
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
338 Example:
339
340 GET dirty_jobs/close/open
341 (receives job 1)
342 GET dirty_jobs/close/open
343 (closes job 1, receives job 2)
344 ...etc...
58746d2 more docs!
Robey Pointer authored
345
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
346
58746d2 more docs!
Robey Pointer authored
347 Server stats
348 ------------
349
66c498b docs!
Robey Pointer authored
350 Global stats reported by kestrel are:
351
352 - `uptime` - seconds the server has been online
353 - `time` - current time in unix epoch
354 - `version` - version string, like "1.2"
355 - `curr_items` - total of items waiting in all queues
356 - `total_itmes` - total of items that have ever been added in this server's
357 lifetime
358 - `bytes` - total byte size of items waiting in all queues
359 - `curr_connections` - current open connections from clients
360 - `total_connections` - total connections that have been opened in this
361 server's lifetime
362 - `cmd_get` - total `GET` requests
363 - `cmd_set` - total `SET` requests
364 - `cmd_peek` - total `GET/peek` requests
365 - `get_hits` - total `GET` requests that received an item
366 - `get_misses` - total `GET` requests on an empty queue
367 - `bytes_read` - total bytes read from clients
368 - `bytes_written` - total bytes written to clients
0d8a507 @gphat Add more statistics along with tests and docs.
gphat authored
369 - `queue_creates` - total number of queues created
7841145 @gphat Expiring queues.
gphat authored
370 - `queue_deletes` - total number of queues deleted (includes expires)
371 - `queue_expires` - total number of queues expires
66c498b docs!
Robey Pointer authored
372
373 For each queue, the following stats are also reported:
374
375 - `items` - items waiting in this queue
376 - `bytes` - total byte size of items waiting in this queue
377 - `total_items` - total items that have been added to this queue in this
378 server's lifetime
379 - `logsize` - byte size of the queue's journal file
380 - `expired_items` - total items that have been expired from this queue in this
381 server's lifetime
382 - `mem_items` - items in this queue that are currently in memory
383 - `mem_bytes` - total byte size of items in this queue that are currently in
384 memory (will always be less than or equal to `max_memory_size` config for
385 the queue)
386 - `age` - time, in milliseconds, that the last item to be fetched from this
387 queue had been waiting; that is, the time between `SET` and `GET`; if the
388 queue is empty, this will always be zero
389 - `discarded` - number of items discarded because the queue was too full
390 - `waiters` - number of clients waiting for an item from this queue (using
391 `GET/t`)
721bf6b document the new stat.
Robey Pointer authored
392 - `open_transactions` - items read with `/open` but not yet confirmed
5e4a681 @gphat Add create_time as a queue stat and document age_msec.
gphat authored
393 - `total_flushes` - total number of times this queue has been flushed
394 - `age_msec` - age of the last item read from the queue
395 - `create_time` - the time that the queue was created (in milliseconds since epoch)
66c498b docs!
Robey Pointer authored
396
397 Kestrel as a library
398 --------------------
399
400 You can use kestrel as a library by just sticking the jar on your classpath.
401 It's a cheap way to get a durable work queue for inter-process or inter-thread
402 communication. Each queue is represented by a `PersistentQueue` object:
403
1531d0d finish cleaning up the guide.
Robey Pointer authored
404 class PersistentQueue(val name: String, persistencePath: String,
405 @volatile var config: QueueConfig, timer: Timer,
406 queueLookup: Option[(String => Option[PersistentQueue])]) {
66c498b docs!
Robey Pointer authored
407
408 and must be initialized before using:
409
410 def setup(): Unit
58746d2 more docs!
Robey Pointer authored
411
66c498b docs!
Robey Pointer authored
412 specifying the path for the journal files (if the queue will be journaled),
1531d0d finish cleaning up the guide.
Robey Pointer authored
413 the name of the queue, a `QueueConfig` object (derived from `QueueBuilder`),
414 a timer for handling timeout reads, and optionally a way to find other named
415 queues (for `expireToQueue` support).
58746d2 more docs!
Robey Pointer authored
416
66c498b docs!
Robey Pointer authored
417 To add an item to a queue:
58746d2 more docs!
Robey Pointer authored
418
1531d0d finish cleaning up the guide.
Robey Pointer authored
419 def add(value: Array[Byte], expiry: Option[Time]): Boolean
58746d2 more docs!
Robey Pointer authored
420
66c498b docs!
Robey Pointer authored
421 It will return `false` if the item was rejected because the queue was full.
0c1b88b lots more docs.
Robey Pointer authored
422
66c498b docs!
Robey Pointer authored
423 Queue items are represented by a case class:
424
1531d0d finish cleaning up the guide.
Robey Pointer authored
425 case class QItem(addTime: Time, expiry: Option[Time], data: Array[Byte], var xid: Int)
66c498b docs!
Robey Pointer authored
426
427 and several operations exist to remove or peek at the head item:
428
429 def peek(): Option[QItem]
430 def remove(): Option[QItem]
431
432 To open a reliable read, set `transaction` true, and later confirm or unremove
433 the item by its `xid`:
434
435 def remove(transaction: Boolean): Option[QItem]
436 def unremove(xid: Int)
437 def confirmRemove(xid: Int)
438
1531d0d finish cleaning up the guide.
Robey Pointer authored
439 You can also asynchronously remove or peek at items using futures.
66c498b docs!
Robey Pointer authored
440
1531d0d finish cleaning up the guide.
Robey Pointer authored
441 def waitRemove(deadline: Option[Time], transaction: Boolean): Future[Option[QItem]]
442 def waitPeek(deadline: Option[Time]): Future[Option[QItem]]
66c498b docs!
Robey Pointer authored
443
444 When done, you should close the queue:
445
446 def close(): Unit
447 def isClosed: Boolean
448
449 Here's a short example:
450
1531d0d finish cleaning up the guide.
Robey Pointer authored
451 var queue = new PersistentQueue("work", "/var/spool/kestrel", config, timer, None)
66c498b docs!
Robey Pointer authored
452 queue.setup()
453
454 // add an item with no expiration:
455 queue.add("hello".getBytes, 0)
456
457 // start to remove it, then back out:
458 val item = queue.remove(true)
459 queue.unremove(item.xid)
460
461 // remove an item with a 500msec timeout, and confirm it:
1531d0d finish cleaning up the guide.
Robey Pointer authored
462 queue.waitRemove(500.milliseconds.fromNow, true)() match {
463 case None =>
464 println("nothing. :(")
465 case Some(item) =>
466 println("got: " + new String(item.data))
467 queue.confirmRemove(item.xid)
66c498b docs!
Robey Pointer authored
468 }
0c1b88b lots more docs.
Robey Pointer authored
469
66c498b docs!
Robey Pointer authored
470 queue.close()
Something went wrong with that request. Please try again.