Skip to content


Subversion checkout URL

You can clone with
Download ZIP
100644 152 lines (102 sloc) 6 KB
0c1b88b lots more docs.
Robey Pointer authored
2 An operational guide to kestrel
3 ===============================
5 Kestrel is a very simple message queue that runs on the JVM and uses the
6 memcache protocol (with some extensions) to talk to clients.
8 A single kestrel server has a set of queues identified by a name, which is
9 also the filename of that queue's journal file (usually in
10 `/var/spool/kestrel`). Each queue is a strictly-ordered FIFO of "items" of
11 binary data. Usually this data is in some serialized format like JSON or
12 ruby's marshal format.
14 A cluster of kestrel servers is like a memcache cluster: the servers don't
15 know about each other, and don't do any cross-communication, so you can add as
16 many as you like. Clients have a list of all servers in the cluster, and pick
17 one at random for each operation. In this way, each queue appears to be spread
18 out across every server in a loose ordering.
21 Configuration
22 -------------
24 All of the per-queue configuration can be set in the global scope of
25 `production.conf`, as a default for all queues, or in the per-queue
26 configuration to override the defaults for a specific queue. You can see an
27 example of this in the default config file.
29 To confirm the current configuration of each queue, send "dump_config" to
30 a server (which can be done over telnet).
32 To reload the config file on a running server, send "reload" the same way.
33 You should immediately see the changes in "dump_config", to confirm.
35 - `max_items` (infinite)
37 Set a hard limit on the number of items this queue can hold. When the queue
38 is full, `discard_old_when_full` dictates the behavior when a client
39 attempts to add another item.
41 - `max_size` (infinite)
43 Set a hard limit on the number of bytes (of data in queued items) this
44 queue can hold. When the queue is full, `discard_old_when_full` dictates
45 the behavior when a client attempts to add another item.
47 - `discard_old_when_full` (false)
49 If this is false, when a queue is full, clients attempting to add another
50 item will get an error. No new items will be accepted. If this is true, old
51 items will be discarded to make room for the new one. This settting has no
52 effect unless at least one of `max_items` or `max_size` is set.
54 - `journal` (true)
56 If false, don't keep a journal file for this queue. When kestrel exits, any
57 remaining contents in the queue will be lost.
59 - `sync_journal` (false)
61 If true, sync the journal file on disk after each write. This is usually
62 not necessary but is available for the paranoid. It will probably reduce
63 the maximum throughput of the server.
65 - `max_journal_size` (16MB)
67 When a journal reaches this size, it will be rolled over to a new file as
68 soon as the queue is empty. The value must be given in bytes.
70 - `max_journal_overflow` (10)
72 If a journal file grows to this many times its desired maximum size, and
73 the total queue contents (in bytes) are smaller than the desired maximum
74 size, the journal file will be rewritten from scratch, to avoid using up
75 all disk space. For example, using the default `max_journal_size` of 16MB
76 and `max_journal_overflow` of 10, if the journal file ever grows beyond
77 160MB (and the queue's contents are less than 16MB), the journal file will
78 be re-written.
80 - `max_journal_size_absolute` (infinite)
82 When a journal reaches this size, it will be rolled over to a new file
83 *no matter how big the queue is*. The value must be given in bytes.
85 - `max_memory_size` (128MB)
87 If a queue's contents grow past this size, only this part will be kept in
88 memory. Newly added items will be written directly to the journal file and
89 read back into memory as the queue is drained. This setting is a release
90 valve to keep a backed-up queue from consuming all memory. The value must
91 be given in bytes.
93 - `max_age` (0 = off)
95 Expiration time (in milliseconds) for items on this queue. Any item that
96 has been sitting on the queue longer than this amount will be discarded.
97 Clients may also attach an expiration time when adding items to a queue,
98 but if the expiration time is longer than `max_age`, `max_age` will be
99 used instead.
102 The journal file
103 ----------------
105 The journal file is the only on-disk storage of a queue's contents, and it's
106 just a sequential record of each add or remove operation that's happened on
107 that queue. When kestrel starts up, it replays each queue's journal to build
108 up the in-memory queue that it uses for client queries.
110 The journal file is rotated in one of three conditions:
112 1. the queue is empty and the journal is larger than `max_journal_size`
114 2. the queue is smaller than `max_journal_size` but the journal is larger
115 than `max_journal_overflow` times `max_journal_size`
117 3. the journal is larger than `max_journal_size_absolute`
119 For example, if `max_journal_size` is 16MB (the default), and
120 `max_journal_overflow` is 10 (also the default), then if the queue is empty
121 and the journal is larger than 16MB, it will be rotated into a new (empty)
122 file. If the queue is smaller than 16MB, but the journal is larger than 160MB,
123 the journal will be rotated to contain just the live items.
125 You can turn the journal off for a queue (`journal` = false) and the queue
126 will exist only in memory. If the server restarts, all enqueued items are
127 lost. You can also force a queue's journal to be sync'd to disk after every
128 write operation (`sync_journal` = true) at a performance cost.
130 If a queue grows past `max_memory_size` bytes (128MB by default), only the
131 first 128MB is kept in memory. The journal is used to track later items, and
132 as items are removed, the journal is played forward to keep 128MB in memory.
133 This is usually known as "read-behind" mode, but Twitter engineers sometimes
134 refer to it as the "square snake" because of the diagram used to brainstorm
135 the implementation. When a queue is in read-behind mode, removing an item will
136 often cause 2 disk operations instead of one: one to record the remove, and
137 one to read an item in from disk to keep 128MB in memory. This is the
138 trade-off to avoid filling memory and crashing the JVM.
142 -- expiration of items
144 -- fanout queues
146 -- non-memcache commands (dump_config, flush)
Something went wrong with that request. Please try again.