Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 163 lines (112 sloc) 6.83 kb
0c1b88b lots more docs.
Robey Pointer authored
1
2 An operational guide to kestrel
3 ===============================
4
5 Kestrel is a very simple message queue that runs on the JVM and uses the
6 memcache protocol (with some extensions) to talk to clients.
7
8 A single kestrel server has a set of queues identified by a name, which is
9 also the filename of that queue's journal file (usually in
10 `/var/spool/kestrel`). Each queue is a strictly-ordered FIFO of "items" of
11 binary data. Usually this data is in some serialized format like JSON or
12 ruby's marshal format.
13
d6199a9 explain queue names.
Robey Pointer authored
14 Generally queue names should be limited to alphanumerics `[A-Za-z0-9]`, dash
15 (`-`) and underline (`_`). In practice, kestrel doesn't enforce any
16 restrictions other than the name can't contain slash (`/`) because that can't
17 be used in filenames, squiggle (`~`) because it's used for temporary files,
18 and dot (`.`) because it's reserved for future use.
19
0c1b88b lots more docs.
Robey Pointer authored
20 A cluster of kestrel servers is like a memcache cluster: the servers don't
21 know about each other, and don't do any cross-communication, so you can add as
22 many as you like. Clients have a list of all servers in the cluster, and pick
23 one at random for each operation. In this way, each queue appears to be spread
8c93d94 more doc improvements.
Robey Pointer authored
24 out across every server, with items in a loose ordering.
0c1b88b lots more docs.
Robey Pointer authored
25
26
27 Configuration
28 -------------
29
30 All of the per-queue configuration can be set in the global scope of
8c93d94 more doc improvements.
Robey Pointer authored
31 `production.conf` as a default for all queues, or in the per-queue
0c1b88b lots more docs.
Robey Pointer authored
32 configuration to override the defaults for a specific queue. You can see an
33 example of this in the default config file.
34
35 To confirm the current configuration of each queue, send "dump_config" to
36 a server (which can be done over telnet).
37
38 To reload the config file on a running server, send "reload" the same way.
39 You should immediately see the changes in "dump_config", to confirm.
40
41 - `max_items` (infinite)
42
43 Set a hard limit on the number of items this queue can hold. When the queue
44 is full, `discard_old_when_full` dictates the behavior when a client
45 attempts to add another item.
46
47 - `max_size` (infinite)
48
49 Set a hard limit on the number of bytes (of data in queued items) this
50 queue can hold. When the queue is full, `discard_old_when_full` dictates
51 the behavior when a client attempts to add another item.
52
53 - `discard_old_when_full` (false)
54
55 If this is false, when a queue is full, clients attempting to add another
56 item will get an error. No new items will be accepted. If this is true, old
57 items will be discarded to make room for the new one. This settting has no
58 effect unless at least one of `max_items` or `max_size` is set.
59
60 - `journal` (true)
61
62 If false, don't keep a journal file for this queue. When kestrel exits, any
63 remaining contents in the queue will be lost.
64
65 - `sync_journal` (false)
66
67 If true, sync the journal file on disk after each write. This is usually
68 not necessary but is available for the paranoid. It will probably reduce
69 the maximum throughput of the server.
70
71 - `max_journal_size` (16MB)
72
73 When a journal reaches this size, it will be rolled over to a new file as
74 soon as the queue is empty. The value must be given in bytes.
75
76 - `max_journal_overflow` (10)
77
78 If a journal file grows to this many times its desired maximum size, and
79 the total queue contents (in bytes) are smaller than the desired maximum
80 size, the journal file will be rewritten from scratch, to avoid using up
81 all disk space. For example, using the default `max_journal_size` of 16MB
82 and `max_journal_overflow` of 10, if the journal file ever grows beyond
83 160MB (and the queue's contents are less than 16MB), the journal file will
84 be re-written.
85
86 - `max_journal_size_absolute` (infinite)
87
88 When a journal reaches this size, it will be rolled over to a new file
89 *no matter how big the queue is*. The value must be given in bytes.
90
91 - `max_memory_size` (128MB)
92
93 If a queue's contents grow past this size, only this part will be kept in
94 memory. Newly added items will be written directly to the journal file and
95 read back into memory as the queue is drained. This setting is a release
96 valve to keep a backed-up queue from consuming all memory. The value must
97 be given in bytes.
98
99 - `max_age` (0 = off)
100
101 Expiration time (in milliseconds) for items on this queue. Any item that
102 has been sitting on the queue longer than this amount will be discarded.
103 Clients may also attach an expiration time when adding items to a queue,
104 but if the expiration time is longer than `max_age`, `max_age` will be
105 used instead.
106
107
108 The journal file
109 ----------------
110
111 The journal file is the only on-disk storage of a queue's contents, and it's
112 just a sequential record of each add or remove operation that's happened on
113 that queue. When kestrel starts up, it replays each queue's journal to build
114 up the in-memory queue that it uses for client queries.
115
116 The journal file is rotated in one of three conditions:
117
118 1. the queue is empty and the journal is larger than `max_journal_size`
119
120 2. the queue is smaller than `max_journal_size` but the journal is larger
121 than `max_journal_overflow` times `max_journal_size`
122
123 3. the journal is larger than `max_journal_size_absolute`
124
125 For example, if `max_journal_size` is 16MB (the default), and
126 `max_journal_overflow` is 10 (also the default), then if the queue is empty
127 and the journal is larger than 16MB, it will be rotated into a new (empty)
128 file. If the queue is smaller than 16MB, but the journal is larger than 160MB,
129 the journal will be rotated to contain just the live items.
130
131 You can turn the journal off for a queue (`journal` = false) and the queue
132 will exist only in memory. If the server restarts, all enqueued items are
133 lost. You can also force a queue's journal to be sync'd to disk after every
134 write operation (`sync_journal` = true) at a performance cost.
135
136 If a queue grows past `max_memory_size` bytes (128MB by default), only the
137 first 128MB is kept in memory. The journal is used to track later items, and
138 as items are removed, the journal is played forward to keep 128MB in memory.
139 This is usually known as "read-behind" mode, but Twitter engineers sometimes
140 refer to it as the "square snake" because of the diagram used to brainstorm
141 the implementation. When a queue is in read-behind mode, removing an item will
142 often cause 2 disk operations instead of one: one to record the remove, and
143 one to read an item in from disk to keep 128MB in memory. This is the
144 trade-off to avoid filling memory and crashing the JVM.
145
146
147
575a57f docs notes.
Robey Pointer authored
148 -- expiration of items (esp: when does it happen)
0c1b88b lots more docs.
Robey Pointer authored
149
150 -- fanout queues
575a57f docs notes.
Robey Pointer authored
151 - names are <master>+<name>: orders, orders+client1, orders+client2
152 - config for all fanout queues comes from the master queue (orders) not orders+foo
153 - uses (k+1)N disk space where k = fanout slaves, N = size of queue
0c1b88b lots more docs.
Robey Pointer authored
154
155
575a57f docs notes.
Robey Pointer authored
156 -- non-memcache commands
8c93d94 more doc improvements.
Robey Pointer authored
157 ("SHUTDOWN", "RELOAD", "FLUSH", "FLUSH_ALL", "DUMP_CONFIG", "DROP")
0c1b88b lots more docs.
Robey Pointer authored
158
8c93d94 more doc improvements.
Robey Pointer authored
159 -- what's in stats? "STATS"
0c1b88b lots more docs.
Robey Pointer authored
160
161
162
Something went wrong with that request. Please try again.