Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 341 lines (264 sloc) 14.798 kb
daf10f0 better readme.
Robey Pointer authored
1
f826e3b beginning of kestrel rename.
Robey Pointer authored
2 Kestrel
3 =======
daf10f0 better readme.
Robey Pointer authored
4
8b0f87b cleanup docs a little more.
Robey Pointer authored
5 Kestrel is based on Blaine Cook's "starling" simple, distributed message
6 queue, with added features and bulletproofing, as well as the scalability
7 offered by actors and the JVM.
daf10f0 better readme.
Robey Pointer authored
8
1ebb032 Michael Wolf fix minor typo
maw authored
9 Each server handles a set of reliable, ordered message queues. When you put a
8b0f87b cleanup docs a little more.
Robey Pointer authored
10 cluster of these servers together, *with no cross communication*, and pick a
11 server at random whenever you do a `set` or `get`, you end up with a reliable,
12 *loosely ordered* message queue.
daf10f0 better readme.
Robey Pointer authored
13
14 In many situations, loose ordering is sufficient. Dropping the requirement on
15 cross communication makes it horizontally scale to infinity and beyond: no
16 multicast, no clustering, no "elections", no coordination at all. No talking!
17 Shhh!
18
8b0f87b cleanup docs a little more.
Robey Pointer authored
19 For more information about what it is and how to use it, check out
90bac3d Switched to a fully qualified URL to fix buggy github linking.
Steve Jenson authored
20 the included [guide](https://github.com/robey/kestrel/blob/master/docs/guide.md).
f826e3b beginning of kestrel rename.
Robey Pointer authored
21
f4ab63a add mailing list info.
Robey Pointer authored
22 Kestrel has a mailing list here: <kestrel-talk@googlegroups.com>
23 http://groups.google.com/group/kestrel-talk
24
0fec041 document flood and packing.
Robey Pointer authored
25 Author's address: Robey Pointer <<robeypointer@gmail.com>>
26
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
27
daf10f0 better readme.
Robey Pointer authored
28 Features
29 --------
30
f826e3b beginning of kestrel rename.
Robey Pointer authored
31 Kestrel is:
daf10f0 better readme.
Robey Pointer authored
32
33 - fast
34
35 It runs on the JVM so it can take advantage of the hard work people have
36 put into java performance.
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
37
daf10f0 better readme.
Robey Pointer authored
38 - small
39
e45676a incorporate matt sanford's feedback.
Robey Pointer authored
40 Currently about 2K lines of scala (including comments), because it relies
2afb9b5 correct casing and fixed a misspelled word
Steve Jenson authored
41 on Apache Mina (a rough equivalent of Danger's ziggurat or Ruby's
f826e3b beginning of kestrel rename.
Robey Pointer authored
42 EventMachine) and actors -- and frankly because Scala is extremely
daf10f0 better readme.
Robey Pointer authored
43 expressive.
44
45 - durable
46
47 Queues are stored in memory for speed, but logged into a journal on disk
48 so that servers can be shutdown or moved without losing any data.
f826e3b beginning of kestrel rename.
Robey Pointer authored
49
50 - reliable
51
52 A client can ask to "tentatively" fetch an item from a queue, and if that
53 client disconnects from kestrel before confirming ownership of the item,
54 the item is handed to another client. In this way, crashing clients don't
55 cause lost messages.
56
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
57
daf10f0 better readme.
Robey Pointer authored
58 Anti-Features
59 -------------
60
f826e3b beginning of kestrel rename.
Robey Pointer authored
61 Kestrel is not:
daf10f0 better readme.
Robey Pointer authored
62
63 - strongly ordered
64
65 While each queue is strongly ordered on each machine, a cluster will
66 appear "loosely ordered" because clients pick a machine at random for
67 each operation. The end result should be "mostly fair".
68
69 - transactional
70
f826e3b beginning of kestrel rename.
Robey Pointer authored
71 This is not a database. Item ownership is transferred with acknowledgement,
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
72 but kestrel does not support grouping multiple operations into an atomic
73 unit.
daf10f0 better readme.
Robey Pointer authored
74
75
8c93d94 more doc improvements.
Robey Pointer authored
76 Building it
77 -----------
78
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
79 Kestrel requires java 6 and sbt 0.7.4. On OS X 10.5, you may have to hard-code
80 an annoying `JAVA_HOME` to use java 6:
daf10f0 better readme.
Robey Pointer authored
81
8c93d94 more doc improvements.
Robey Pointer authored
82 $ export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home
f31c8f0 mention the java 6 requirement.
Robey Pointer authored
83
daf10f0 better readme.
Robey Pointer authored
84 Building from source is easy:
85
6baebef remove old ant files. update README to use sbt instead.
Robey Pointer authored
86 $ sbt clean update package-dist
8c93d94 more doc improvements.
Robey Pointer authored
87
daf10f0 better readme.
Robey Pointer authored
88 Scala libraries and dependencies will be downloaded from maven repositories
89 the first time you do a build. The finished distribution will be in `dist`.
90
91
8c93d94 more doc improvements.
Robey Pointer authored
92 Running it
93 ----------
94
95 You can run kestrel by hand via:
96
97 $ java -jar ./dist/kestrel-VERSION/kestrel-VERSION.jar
98
99 To run in development mode (using `development.conf` instead of
100 `production.conf`), add a `stage` variable:
101
102 $ java -Dstage=development -jar ./dist/kestrel-VERSION/kestrel-VERSION.jar
103
104 When running it as a server, a startup script is provided in
105 `dist/kestrel-VERSION/scripts/kestrel.sh`. The script assumes you have
106 `daemon`, a standard daemonizer for Linux, but also available
107 [here](http://libslack.org/daemon/) for all common unix platforms.
108
109 The created archive `kestrel-VERSION.tar.bz2` can be expanded into a place
110 like `/usr/local` (or wherever you like) and executed within its own folder as
111 a self-contained package. All dependent jars are included, and the startup
b900ea8 document config options for queue.
Robey Pointer authored
112 script loads things from relative paths.
113
114 The default configuration puts logfiles into `/var/log/kestrel/` and queue
115 journal files into `/var/spool/kestrel/`.
116
8c93d94 more doc improvements.
Robey Pointer authored
117 The startup script logs extensive GC information to a file named `stdout` in
118 the log folder. If kestrel has problems starting up (before it can initialize
119 logging), it will usually appear in `error` in the same folder.
120
b900ea8 document config options for queue.
Robey Pointer authored
121
122 Configuration
123 -------------
124
0c1b88b lots more docs.
Robey Pointer authored
125 Queue configuration is described in detail in `docs/guide.md` (an operational
70ef6d0 add some more docs, and make the guide refer to them.
Robey Pointer authored
126 guide). Scala docs for the config variables are here:
127 http://robey.github.com/kestrel/doc/main/api/net/lag/kestrel/config/KestrelConfig.html
74f21d8 document expiration_timer_frequency_seconds, max_item_size, and move_exp...
Robey Pointer authored
128
daf10f0 better readme.
Robey Pointer authored
129
130 Performance
131 -----------
132
e25e6bb clean up put-many and add it to the docs. (1 down, 3 to go.)
Robey Pointer authored
133 Several performance tests are included. To run them, first start up a kestrel instance
134 locally.
135
136 $ sbt clean update package-dist
137 $ VERSION="2.1.0-SNAPSHOT"
138 $ java -server -Xmx1024m -Dstage=development -jar ./dist/kestrel-$VERSION/kestrel-$VERSION.jar
139
140 ## Put-many
141
142 This test just spams a kestrel server with "put" operations, to see how
143 quickly it can absorb and journal them.
144
145 $ sbt "put-many --help"
146 usage: put-many [options]
147 spam items into kestrel
148
149 options:
150 -c CLIENTS
151 use CLIENTS concurrent clients (default: 100)
152 -n ITEMS
153 put ITEMS items into the queue (default: 10000)
154 -b BYTES
155 put BYTES per queue item (default: 1024)
156
157 A sample run on a 2010 MacBook Pro:
158
159 [info] == put-many ==
160 [info] Running net.lag.kestrel.load.PutMany -n 1000000
161 Finished in 64921 msec (64.9 usec/put throughput).
162 Transactions: min=95.00; max=528107.00 524847.00 521780.00;
163 median=3433.00; average=5551.77 usec
164 Transactions distribution: 5.00%=810.00 10.00%=1526.00 25.00%=2414.00
165 50.00%=3433.00 75.00%=4851.00 90.00%=6933.00 95.00%=9145.00
166 99.00%=59133.00 99.90%=208001.00 99.99%=505030.00
167 [info] == put-many ==
168
8486640 update and document many-clients. (2/4!)
Robey Pointer authored
169 ## Many-clients
170
171 This test has one producer that trickles out one item at a time, and a pile of
172 consumers fighting for each item. It usually takes exactly as long as the
173 number of items times the delay, but is useful as a validation test to make
174 sure kestrel works as advertised without blowing up.
175
176 $ sbt "many-clients --help"
177 usage: many-clients [options]
178 spin up N clients and have them do timeout reads on a queue while a
179 single producer trickles out.
180
181 options:
182 -s MILLESCONDS
183 sleep MILLISECONDS between puts (default: 100)
184 -n ITEMS
185 put ITEMS total items into the queue (default: 100)
186 -c CLIENTS
187 use CLIENTS consumers (default: 100)
188
189 A sample run on a 2010 MacBook Pro:
190
191 [info] == many-clients ==
192 [info] Running net.lag.kestrel.load.ManyClients
193 Finished in 11104 msec.
194 [info] == many-clients ==
195
0fec041 document flood and packing.
Robey Pointer authored
196 ## Flood
8486640 update and document many-clients. (2/4!)
Robey Pointer authored
197
0fec041 document flood and packing.
Robey Pointer authored
198 This test starts up one producer and one consumer, and just floods items
199 through kestrel as fast as it can.
8486640 update and document many-clients. (2/4!)
Robey Pointer authored
200
0fec041 document flood and packing.
Robey Pointer authored
201 $ sbt "flood --help"
202 usage: flood [options]
203 spin up a producer and consumer and flood N items through kestrel
8486640 update and document many-clients. (2/4!)
Robey Pointer authored
204
0fec041 document flood and packing.
Robey Pointer authored
205 options:
206 -n ITEMS
207 put ITEMS items into the queue (default: 10000)
208 -k KILOBYTES
209 put KILOBYTES per queue item (default: 1)
e25e6bb clean up put-many and add it to the docs. (1 down, 3 to go.)
Robey Pointer authored
210
0fec041 document flood and packing.
Robey Pointer authored
211 A sample run on a 2010 MacBook Pro:
daf10f0 better readme.
Robey Pointer authored
212
0fec041 document flood and packing.
Robey Pointer authored
213 [info] == flood ==
214 [info] Running net.lag.kestrel.load.Flood -n 100000
215 flood: 100000 items of 1kB
216 Finished in 16834 msec (168.3 usec/put throughput).
217 Consumer spun 1 times in misses.
daf10f0 better readme.
Robey Pointer authored
218
0fec041 document flood and packing.
Robey Pointer authored
219 ## Packing
daf10f0 better readme.
Robey Pointer authored
220
0fec041 document flood and packing.
Robey Pointer authored
221 This test starts up one producer and one consumer, seeds the queue with a
222 bunch of items to cause it to fall behind, then does cycles of flooding items
223 through the queue, separated by pauses. It's meant to test kestrel's behavior
224 with a queue that's fallen behind and *stays* behind indefinitely, to make
225 sure the journal files are packed periodically without affecting performance
226 too badly.
daf10f0 better readme.
Robey Pointer authored
227
0fec041 document flood and packing.
Robey Pointer authored
228 $ sbt "packing --help"
229 usage: packing [options]
230 spin up a producer and consumer, write N items, then do read/write cycles
231 with pauses
0e4d572 add some recent distro stats.
Robey Pointer authored
232
0fec041 document flood and packing.
Robey Pointer authored
233 options:
234 -q NAME
235 use named queue (default: spam)
236 -n ITEMS
237 put ITEMS items into the queue (default: 25000)
238 -k KILOBYTES
239 put KILOBYTES per queue item (default: 1)
240 -t SECONDS
241 pause SECONDS between cycles (default: 1)
242 -c CYCLES
243 do read/writes CYCLES times (default: 100)
244 -x
245 use transactions when fetching
0e4d572 add some recent distro stats.
Robey Pointer authored
246
0fec041 document flood and packing.
Robey Pointer authored
247 A sample run on a 2010 MacBook Pro:
0e4d572 add some recent distro stats.
Robey Pointer authored
248
0fec041 document flood and packing.
Robey Pointer authored
249 [info] == packing ==
250 [info] Running net.lag.kestrel.load.JournalPacking -c 10 -q small
251 packing: 25000 items of 1kB with 1 second pauses
252 Wrote 25000 items starting at 0.
253 cycle: 1
254 Wrote 25000 items starting at 25000.
255 Read 25000 items in 5402 msec. Consumer spun 0 times in misses.
256 cycle: 2
257 Wrote 25000 items starting at 50000.
258 Read 25000 items in 5395 msec. Consumer spun 0 times in misses.
259 cycle: 3
260 Wrote 25000 items starting at 75000.
261 Read 25000 items in 4584 msec. Consumer spun 0 times in misses.
262 cycle: 4
263 Wrote 25000 items starting at 100000.
264 Read 25000 items in 4455 msec. Consumer spun 0 times in misses.
265 cycle: 5
266 Wrote 25000 items starting at 125000.
267 Read 25000 items in 4742 msec. Consumer spun 0 times in misses.
268 cycle: 6
269 Wrote 25000 items starting at 150000.
270 Read 25000 items in 4468 msec. Consumer spun 0 times in misses.
271 cycle: 7
272 Wrote 25000 items starting at 175000.
273 Read 25000 items in 5127 msec. Consumer spun 0 times in misses.
274 cycle: 8
275 Wrote 25000 items starting at 200000.
276 Read 25000 items in 4357 msec. Consumer spun 0 times in misses.
277 cycle: 9
278 Wrote 25000 items starting at 225000.
279 Read 25000 items in 4500 msec. Consumer spun 0 times in misses.
280 cycle: 10
281 Wrote 25000 items starting at 250000.
282 Read 25000 items in 4558 msec. Consumer spun 0 times in misses.
283 Read 25000 items in 3141 msec. Consumer spun 0 times in misses.
284 [info] == packing ==
285
286 You can see the journals being packed in the kestrel log:
287
288 INF [20110405-20:36:57.420] kestrel: Setting up queue small: maxItems=2147483647 maxSize=134217728.bytes maxItemSize=922
289 3372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=16777216.bytes maxJournalSize=1342177
290 28.bytes discardOldWhenFull=true keepJournal=true syncJournal=never expireToQueue=None maxExpireSweep=2147483647 fanoutO
291 nly=false
292 INF [20110405-20:36:57.421] kestrel: Replaying transaction journal for 'small'
293 INF [20110405-20:36:57.422] kestrel: No transaction journal for 'small'; starting with empty queue.
294 INF [20110405-20:36:57.422] kestrel: Finished transaction journal for 'small' (0 items, 0 bytes) xid=0
295 INF [20110405-20:36:59.779] kestrel: Rotating journal file for 'small' (qsize=16440320)
296 INF [20110405-20:36:59.852] kestrel: Dropping to read-behind for queue 'small' (16.0 MiB)
297 INF [20110405-20:37:02.032] kestrel: Rotating journal file for 'small' (qsize=29139968)
298 INF [20110405-20:37:04.583] kestrel: Rotating journal file for 'small' (qsize=35066880)
299 INF [20110405-20:37:05.005] kestrel: Read-behind on 'small' moving from file small.1302061022051 to small.1302061024673
300 INF [20110405-20:37:08.547] kestrel: Read-behind on 'small' moving from file small.1302061024673 to small
301 INF [20110405-20:37:09.553] kestrel: Rotating journal file for 'small' (qsize=27975680)
302 INF [20110405-20:37:12.412] kestrel: Read-behind on 'small' moving from file small.1302061029571 to small
303 INF [20110405-20:37:14.511] kestrel: Rotating journal file for 'small' (qsize=26700800)
304 INF [20110405-20:37:16.384] kestrel: Read-behind on 'small' moving from file small.1302061034588 to small
305 INF [20110405-20:37:17.122] kestrel: Rotating journal file for 'small' (qsize=29371392)
306 INF [20110405-20:37:20.164] kestrel: Read-behind on 'small' moving from file small.1302061037149 to small
307 INF [20110405-20:37:21.410] kestrel: Rotating journal file for 'small' (qsize=26664960)
308 INF [20110405-20:37:23.113] kestrel: Read-behind on 'small' moving from file small.1302061041427 to small
309 INF [20110405-20:37:25.302] kestrel: Rotating journal file for 'small' (qsize=26168320)
310 INF [20110405-20:37:27.118] kestrel: Read-behind on 'small' moving from file small.1302061045321 to small
311 INF [20110405-20:37:27.119] kestrel: Rewriting journal file from checkpoint for 'small' (qsize=27889664)
312 INF [20110405-20:37:27.129] kestrel: Packing journals for 'small': small.1302061019805, small.1302061022051, small.13020
313 61024673, small.1302061029571, small.1302061034588, small.1302061037149, small.1302061041427, small.1302061045321
314 INF [20110405-20:37:27.635] kestrel: Packing 'small' -- erasing old files.
315 INF [20110405-20:37:27.646] kestrel: Packing 'small' done: small.1302061045321, small
316 INF [20110405-20:37:28.115] kestrel: Rotating journal file for 'small' (qsize=28761088)
317 INF [20110405-20:37:31.108] kestrel: Read-behind on 'small' moving from file small.1302061048143 to small
318 INF [20110405-20:37:32.202] kestrel: Rotating journal file for 'small' (qsize=27242496)
319 INF [20110405-20:37:34.048] kestrel: Read-behind on 'small' moving from file small.1302061052221 to small
320 INF [20110405-20:37:36.255] kestrel: Rotating journal file for 'small' (qsize=25759744)
321 INF [20110405-20:37:38.433] kestrel: Read-behind on 'small' moving from file small.1302061056360 to small
322 INF [20110405-20:37:39.550] kestrel: Rotating journal file for 'small' (qsize=27325440)
323 INF [20110405-20:37:42.266] kestrel: Read-behind on 'small' moving from file small.1302061059646 to small
324 INF [20110405-20:37:43.464] kestrel: Rotating journal file for 'small' (qsize=26256384)
325 INF [20110405-20:37:45.110] kestrel: Read-behind on 'small' moving from file small.1302061063469 to small
326 INF [20110405-20:37:46.110] kestrel: Rotating journal file for 'small' (qsize=27487232)
327 INF [20110405-20:37:48.928] kestrel: Read-behind on 'small' moving from file small.1302061066128 to small
328 INF [20110405-20:37:49.875] kestrel: Rotating journal file for 'small' (qsize=28101632)
329 INF [20110405-20:37:51.801] kestrel: Read-behind on 'small' moving from file small.1302061069893 to small
330 INF [20110405-20:37:51.801] kestrel: Rewriting journal file from checkpoint for 'small' (qsize=26379264)
331 INF [20110405-20:37:51.804] kestrel: Packing journals for 'small': small.1302061045321, small.1302061048143, small.13020
332 61052221, small.1302061056360, small.1302061059646, small.1302061063469, small.1302061066128, small.1302061069893
333 INF [20110405-20:37:52.237] kestrel: Packing 'small' -- erasing old files.
334 INF [20110405-20:37:52.246] kestrel: Packing 'small' done: small.1302061069893, small
335 INF [20110405-20:37:54.012] kestrel: Rotating journal file for 'small' (qsize=26510336)
336 INF [20110405-20:37:55.808] kestrel: Read-behind on 'small' moving from file small.1302061074039 to small
337 INF [20110405-20:37:56.594] kestrel: Rotating journal file for 'small' (qsize=29006848)
338 INF [20110405-20:37:59.363] kestrel: Read-behind on 'small' moving from file small.1302061076614 to small
339 INF [20110405-20:37:59.731] kestrel: Coming out of read-behind for queue 'small'
daf10f0 better readme.
Robey Pointer authored
340
Something went wrong with that request. Please try again.