Permalink
Browse files

Merge remote branch 'robey/master'

Conflicts:
	src/main/scala/net/lag/kestrel/KestrelHandler.scala
  • Loading branch information...
Jason Jackson
Jason Jackson committed Sep 7, 2011
2 parents 100d4d9 + b2baaa8 commit 2ecde3ac42a53bda22f1a72b336e6dc486e0f153
Showing with 2,006 additions and 999 deletions.
  1. +7 −0 .gitignore
  2. +216 −75 README.md
  3. +0 −62 bin/net/lag/TestHelper.scala
  4. +13 −6 config/development.scala
  5. +4 −3 config/production.scala
  6. +65 −112 docs/guide.md
  7. +104 −0 docs/journal_format.md
  8. +0 −2 docs/text_protocol.md
  9. +2 −2 project/build.properties
  10. +24 −16 project/build/KestrelProject.scala
  11. +19 −4 project/plugins/Plugins.scala
  12. +4 −0 project/release.properties
  13. +63 −0 src/main/scala/net/lag/kestrel/DeadlineWaitQueue.scala
  14. +263 −158 src/main/scala/net/lag/kestrel/Journal.scala
  15. +17 −9 src/main/scala/net/lag/kestrel/JournalPacker.scala
  16. +17 −12 src/main/scala/net/lag/kestrel/Kestrel.scala
  17. +14 −18 src/main/scala/net/lag/kestrel/KestrelHandler.scala
  18. +10 −23 src/main/scala/net/lag/kestrel/MemcacheHandler.scala
  19. +3 −2 src/main/scala/net/lag/kestrel/NettyHandler.scala
  20. +79 −0 src/main/scala/net/lag/kestrel/PeriodicSyncFile.scala
  21. +126 −142 src/main/scala/net/lag/kestrel/PersistentQueue.scala
  22. +5 −6 src/main/scala/net/lag/kestrel/QItem.scala
  23. +18 −48 src/main/scala/net/lag/kestrel/QueueCollection.scala
  24. +16 −3 src/main/scala/net/lag/kestrel/TextHandler.scala
  25. +100 −40 src/main/scala/net/lag/kestrel/config/KestrelConfig.scala
  26. +98 −53 src/main/scala/net/lag/kestrel/tools/QDumper.scala
  27. +220 −0 src/scripts/kcluster
  28. +25 −0 src/test/scala/net/lag/kestrel/DumpJournal.scala
  29. +89 −7 src/test/scala/net/lag/kestrel/JournalSpec.scala
  30. +26 −26 src/test/scala/net/lag/kestrel/KestrelHandlerSpec.scala
  31. +84 −50 src/test/scala/net/lag/kestrel/PersistentQueueSpec.scala
  32. +28 −40 src/test/scala/net/lag/kestrel/QueueCollectionSpec.scala
  33. +3 −4 src/test/scala/net/lag/kestrel/ReadBehindSpec.scala
  34. +2 −2 src/test/scala/net/lag/kestrel/ServerSpec.scala
  35. +3 −1 src/test/scala/net/lag/kestrel/TestLogging.scala
  36. +95 −34 src/test/scala/net/lag/kestrel/TextHandlerSpec.scala
  37. +24 −7 src/test/scala/net/lag/kestrel/load/Flood.scala
  38. +25 −6 src/test/scala/net/lag/kestrel/load/JournalPacking.scala
  39. +43 −16 src/test/scala/net/lag/kestrel/load/ManyClients.scala
  40. +52 −10 src/test/scala/net/lag/kestrel/load/PutMany.scala
View
@@ -12,3 +12,10 @@ project/plugins/src_managed/
project/plugins/project/
lib_managed/
ignore/
+*.iml
+.idea/
+.classpath
+.project
+.scala_dependencies
+.settings/
+go
View
291 README.md
@@ -17,11 +17,13 @@ multicast, no clustering, no "elections", no coordination at all. No talking!
Shhh!
For more information about what it is and how to use it, check out
-the included [guide](docs/guide.md).
+the included [guide](https://github.com/robey/kestrel/blob/master/docs/guide.md).
Kestrel has a mailing list here: <kestrel-talk@googlegroups.com>
http://groups.google.com/group/kestrel-talk
+Author's address: Robey Pointer <<robeypointer@gmail.com>>
+
Features
--------
@@ -32,7 +34,7 @@ Kestrel is:
It runs on the JVM so it can take advantage of the hard work people have
put into java performance.
-
+
- small
Currently about 2K lines of scala (including comments), because it relies
@@ -67,16 +69,15 @@ Kestrel is not:
- transactional
This is not a database. Item ownership is transferred with acknowledgement,
- but kestrel does not support multiple outstanding operations, and treats
- each enqueued item as an atomic unit.
+ but kestrel does not support grouping multiple operations into an atomic
+ unit.
Building it
-----------
-Kestrel requires java 6 (for JMX support) and ant 1.7. If you see an error
-about missing JMX classes, it usually means you're building with java 5. On a
-mac, you may have to hard-code an annoying `JAVA_HOME` to use java 6:
+Kestrel requires java 6 and sbt 0.7.4. On OS X 10.5, you may have to hard-code
+an annoying `JAVA_HOME` to use java 6:
$ export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home
@@ -122,78 +123,218 @@ Configuration
-------------
Queue configuration is described in detail in `docs/guide.md` (an operational
-guide). There are a few global config options that should be self-explanatory:
-
-- `host`
-
- Host to accept connections on.
-
-- `port`
-
- Port to listen on. 22133 is the standard.
-
-- `timeout`
-
- Seconds after which an idle client is disconnected, or 0 to have no idle
- timeout.
-
-- `queue_path`
-
- The folder to store queue journal files in. Each queue (and each client of
- a fanout queue) gets its own file here.
-
-- `log`
-
- Logfile configuration, as described in configgy.
-
-- `expiration_timer_frequency_seconds`
-
- Frequency (in seconds) that a timer thread should scan active queues for
- expired items. By default, this is off (0) and no automatic expiration
- scanning happens; instead, items expire when a client does a `SET` or `GET`
- on a queue. When this is set, a background thread will periodically flush
- expired items from the head of every queue.
+guide). Scala docs for the config variables are here:
+http://robey.github.com/kestrel/doc/main/api/net/lag/kestrel/config/KestrelConfig.html
Performance
-----------
-All of the below timings are on my 2GHz 2006-model macbook pro.
-
-Since starling uses eventmachine in a single-thread single-process form, it
-has similar results for all access types (and will never use more than one
-core).
-
- ========= ================= ==========
- # Clients Pushes per client Total time
- ========= ================= ==========
- 1 10,000 3.8s
- 10 1,000 2.9s
- 100 100 3.1s
- ========= ================= ==========
-
-Kestrel uses N+1 I/O processor threads (where N = the number of available CPU
-cores), and a pool of worker threads for handling actor events. Therefore it
-handles more poorly for small numbers of heavy-use clients, and better for
-large numbers of clients.
-
- ========= ================= ==========
- # Clients Pushes per client Total time
- ========= ================= ==========
- 1 10,000 3.8s
- 10 1,000 2.4s
- 100 100 1.6s
- ========= ================= ==========
-
-A single-threaded set of 5 million puts gives a fair idea of throughput
-distribution, this time on a 2.5GHz 2008-model macbook pro:
-
- $ ant -f tests.xml put-many-1 -Ditems=5000000
- [java] Finished in 1137250 msec (227.5 usec/put throughput).
- [java] Transactions: min=106.00; max=108581.00 91335.00 60721.00; median=153.00; average=201.14 usec
- [java] Transactions distribution: 5.00%=129.00 10.00%=134.00 25.00%=140.00 50.00%=153.00 75.00%=177.00 90.00%=251.00 95.00%=345.00 99.00%=586.00 99.90%=5541.00 99.99%=26910.00
-
-This works out to about 3.23MB/sec (over loopback) and about 4400 puts/sec.
+Several performance tests are included. To run them, first start up a kestrel instance
+locally.
+ $ sbt clean update package-dist
+ $ VERSION="2.1.0-SNAPSHOT"
+ $ java -server -Xmx1024m -Dstage=development -jar ./dist/kestrel-$VERSION/kestrel-$VERSION.jar
+
+## Put-many
+
+This test just spams a kestrel server with "put" operations, to see how
+quickly it can absorb and journal them.
+
+ $ sbt "put-many --help"
+ usage: put-many [options]
+ spam items into kestrel
+
+ options:
+ -c CLIENTS
+ use CLIENTS concurrent clients (default: 100)
+ -n ITEMS
+ put ITEMS items into the queue (default: 10000)
+ -b BYTES
+ put BYTES per queue item (default: 1024)
+
+A sample run on a 2010 MacBook Pro:
+
+ [info] == put-many ==
+ [info] Running net.lag.kestrel.load.PutMany -n 1000000
+ Finished in 64921 msec (64.9 usec/put throughput).
+ Transactions: min=95.00; max=528107.00 524847.00 521780.00;
+ median=3433.00; average=5551.77 usec
+ Transactions distribution: 5.00%=810.00 10.00%=1526.00 25.00%=2414.00
+ 50.00%=3433.00 75.00%=4851.00 90.00%=6933.00 95.00%=9145.00
+ 99.00%=59133.00 99.90%=208001.00 99.99%=505030.00
+ [info] == put-many ==
+
+## Many-clients
+
+This test has one producer that trickles out one item at a time, and a pile of
+consumers fighting for each item. It usually takes exactly as long as the
+number of items times the delay, but is useful as a validation test to make
+sure kestrel works as advertised without blowing up.
+
+ $ sbt "many-clients --help"
+ usage: many-clients [options]
+ spin up N clients and have them do timeout reads on a queue while a
+ single producer trickles out.
+
+ options:
+ -s MILLESCONDS
+ sleep MILLISECONDS between puts (default: 100)
+ -n ITEMS
+ put ITEMS total items into the queue (default: 100)
+ -c CLIENTS
+ use CLIENTS consumers (default: 100)
+
+A sample run on a 2010 MacBook Pro:
+
+ [info] == many-clients ==
+ [info] Running net.lag.kestrel.load.ManyClients
+ Finished in 11104 msec.
+ [info] == many-clients ==
+
+## Flood
+
+This test starts up one producer and one consumer, and just floods items
+through kestrel as fast as it can.
+
+ $ sbt "flood --help"
+ usage: flood [options]
+ spin up a producer and consumer and flood N items through kestrel
+
+ options:
+ -n ITEMS
+ put ITEMS items into the queue (default: 10000)
+ -k KILOBYTES
+ put KILOBYTES per queue item (default: 1)
+
+A sample run on a 2010 MacBook Pro:
+
+ [info] == flood ==
+ [info] Running net.lag.kestrel.load.Flood -n 100000
+ flood: 100000 items of 1kB
+ Finished in 16834 msec (168.3 usec/put throughput).
+ Consumer spun 1 times in misses.
+
+## Packing
+
+This test starts up one producer and one consumer, seeds the queue with a
+bunch of items to cause it to fall behind, then does cycles of flooding items
+through the queue, separated by pauses. It's meant to test kestrel's behavior
+with a queue that's fallen behind and *stays* behind indefinitely, to make
+sure the journal files are packed periodically without affecting performance
+too badly.
+
+ $ sbt "packing --help"
+ usage: packing [options]
+ spin up a producer and consumer, write N items, then do read/write cycles
+ with pauses
+
+ options:
+ -q NAME
+ use named queue (default: spam)
+ -n ITEMS
+ put ITEMS items into the queue (default: 25000)
+ -k KILOBYTES
+ put KILOBYTES per queue item (default: 1)
+ -t SECONDS
+ pause SECONDS between cycles (default: 1)
+ -c CYCLES
+ do read/writes CYCLES times (default: 100)
+ -x
+ use transactions when fetching
+
+A sample run on a 2010 MacBook Pro:
+
+ [info] == packing ==
+ [info] Running net.lag.kestrel.load.JournalPacking -c 10 -q small
+ packing: 25000 items of 1kB with 1 second pauses
+ Wrote 25000 items starting at 0.
+ cycle: 1
+ Wrote 25000 items starting at 25000.
+ Read 25000 items in 5402 msec. Consumer spun 0 times in misses.
+ cycle: 2
+ Wrote 25000 items starting at 50000.
+ Read 25000 items in 5395 msec. Consumer spun 0 times in misses.
+ cycle: 3
+ Wrote 25000 items starting at 75000.
+ Read 25000 items in 4584 msec. Consumer spun 0 times in misses.
+ cycle: 4
+ Wrote 25000 items starting at 100000.
+ Read 25000 items in 4455 msec. Consumer spun 0 times in misses.
+ cycle: 5
+ Wrote 25000 items starting at 125000.
+ Read 25000 items in 4742 msec. Consumer spun 0 times in misses.
+ cycle: 6
+ Wrote 25000 items starting at 150000.
+ Read 25000 items in 4468 msec. Consumer spun 0 times in misses.
+ cycle: 7
+ Wrote 25000 items starting at 175000.
+ Read 25000 items in 5127 msec. Consumer spun 0 times in misses.
+ cycle: 8
+ Wrote 25000 items starting at 200000.
+ Read 25000 items in 4357 msec. Consumer spun 0 times in misses.
+ cycle: 9
+ Wrote 25000 items starting at 225000.
+ Read 25000 items in 4500 msec. Consumer spun 0 times in misses.
+ cycle: 10
+ Wrote 25000 items starting at 250000.
+ Read 25000 items in 4558 msec. Consumer spun 0 times in misses.
+ Read 25000 items in 3141 msec. Consumer spun 0 times in misses.
+ [info] == packing ==
+
+You can see the journals being packed in the kestrel log:
+
+ INF [20110405-20:36:57.420] kestrel: Setting up queue small: maxItems=2147483647 maxSize=134217728.bytes maxItemSize=922
+ 3372036854775807.bytes maxAge=None defaultJournalSize=16777216.bytes maxMemorySize=16777216.bytes maxJournalSize=1342177
+ 28.bytes discardOldWhenFull=true keepJournal=true syncJournal=never expireToQueue=None maxExpireSweep=2147483647 fanoutO
+ nly=false
+ INF [20110405-20:36:57.421] kestrel: Replaying transaction journal for 'small'
+ INF [20110405-20:36:57.422] kestrel: No transaction journal for 'small'; starting with empty queue.
+ INF [20110405-20:36:57.422] kestrel: Finished transaction journal for 'small' (0 items, 0 bytes) xid=0
+ INF [20110405-20:36:59.779] kestrel: Rotating journal file for 'small' (qsize=16440320)
+ INF [20110405-20:36:59.852] kestrel: Dropping to read-behind for queue 'small' (16.0 MiB)
+ INF [20110405-20:37:02.032] kestrel: Rotating journal file for 'small' (qsize=29139968)
+ INF [20110405-20:37:04.583] kestrel: Rotating journal file for 'small' (qsize=35066880)
+ INF [20110405-20:37:05.005] kestrel: Read-behind on 'small' moving from file small.1302061022051 to small.1302061024673
+ INF [20110405-20:37:08.547] kestrel: Read-behind on 'small' moving from file small.1302061024673 to small
+ INF [20110405-20:37:09.553] kestrel: Rotating journal file for 'small' (qsize=27975680)
+ INF [20110405-20:37:12.412] kestrel: Read-behind on 'small' moving from file small.1302061029571 to small
+ INF [20110405-20:37:14.511] kestrel: Rotating journal file for 'small' (qsize=26700800)
+ INF [20110405-20:37:16.384] kestrel: Read-behind on 'small' moving from file small.1302061034588 to small
+ INF [20110405-20:37:17.122] kestrel: Rotating journal file for 'small' (qsize=29371392)
+ INF [20110405-20:37:20.164] kestrel: Read-behind on 'small' moving from file small.1302061037149 to small
+ INF [20110405-20:37:21.410] kestrel: Rotating journal file for 'small' (qsize=26664960)
+ INF [20110405-20:37:23.113] kestrel: Read-behind on 'small' moving from file small.1302061041427 to small
+ INF [20110405-20:37:25.302] kestrel: Rotating journal file for 'small' (qsize=26168320)
+ INF [20110405-20:37:27.118] kestrel: Read-behind on 'small' moving from file small.1302061045321 to small
+ INF [20110405-20:37:27.119] kestrel: Rewriting journal file from checkpoint for 'small' (qsize=27889664)
+ INF [20110405-20:37:27.129] kestrel: Packing journals for 'small': small.1302061019805, small.1302061022051, small.13020
+ 61024673, small.1302061029571, small.1302061034588, small.1302061037149, small.1302061041427, small.1302061045321
+ INF [20110405-20:37:27.635] kestrel: Packing 'small' -- erasing old files.
+ INF [20110405-20:37:27.646] kestrel: Packing 'small' done: small.1302061045321, small
+ INF [20110405-20:37:28.115] kestrel: Rotating journal file for 'small' (qsize=28761088)
+ INF [20110405-20:37:31.108] kestrel: Read-behind on 'small' moving from file small.1302061048143 to small
+ INF [20110405-20:37:32.202] kestrel: Rotating journal file for 'small' (qsize=27242496)
+ INF [20110405-20:37:34.048] kestrel: Read-behind on 'small' moving from file small.1302061052221 to small
+ INF [20110405-20:37:36.255] kestrel: Rotating journal file for 'small' (qsize=25759744)
+ INF [20110405-20:37:38.433] kestrel: Read-behind on 'small' moving from file small.1302061056360 to small
+ INF [20110405-20:37:39.550] kestrel: Rotating journal file for 'small' (qsize=27325440)
+ INF [20110405-20:37:42.266] kestrel: Read-behind on 'small' moving from file small.1302061059646 to small
+ INF [20110405-20:37:43.464] kestrel: Rotating journal file for 'small' (qsize=26256384)
+ INF [20110405-20:37:45.110] kestrel: Read-behind on 'small' moving from file small.1302061063469 to small
+ INF [20110405-20:37:46.110] kestrel: Rotating journal file for 'small' (qsize=27487232)
+ INF [20110405-20:37:48.928] kestrel: Read-behind on 'small' moving from file small.1302061066128 to small
+ INF [20110405-20:37:49.875] kestrel: Rotating journal file for 'small' (qsize=28101632)
+ INF [20110405-20:37:51.801] kestrel: Read-behind on 'small' moving from file small.1302061069893 to small
+ INF [20110405-20:37:51.801] kestrel: Rewriting journal file from checkpoint for 'small' (qsize=26379264)
+ INF [20110405-20:37:51.804] kestrel: Packing journals for 'small': small.1302061045321, small.1302061048143, small.13020
+ 61052221, small.1302061056360, small.1302061059646, small.1302061063469, small.1302061066128, small.1302061069893
+ INF [20110405-20:37:52.237] kestrel: Packing 'small' -- erasing old files.
+ INF [20110405-20:37:52.246] kestrel: Packing 'small' done: small.1302061069893, small
+ INF [20110405-20:37:54.012] kestrel: Rotating journal file for 'small' (qsize=26510336)
+ INF [20110405-20:37:55.808] kestrel: Read-behind on 'small' moving from file small.1302061074039 to small
+ INF [20110405-20:37:56.594] kestrel: Rotating journal file for 'small' (qsize=29006848)
+ INF [20110405-20:37:59.363] kestrel: Read-behind on 'small' moving from file small.1302061076614 to small
+ INF [20110405-20:37:59.731] kestrel: Coming out of read-behind for queue 'small'
-Robey Pointer <<robeypointer@gmail.com>>
Oops, something went wrong.

0 comments on commit 2ecde3a

Please sign in to comment.