Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Browse files

readable readme, and license

  • Loading branch information...
commit d4e19f5516eb4f1354c126fbaec076bc95bc7750 1 parent 450bf65
@erikfrey erikfrey authored
Showing with 62 additions and 57 deletions.
  1. +14 −0 LICENSE
  2. +48 −57
@@ -0,0 +1,14 @@
+ Copyright 2012 Wavii, Inc.
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ See the License for the specific language governing permissions and
+ limitations under the License.
@@ -1,74 +1,65 @@
+# Darner
-Darner is a very simple message queue built on [boost::asio]( and
-[leveldb]( It supports the memcache protocol.
+Darner is a very simple message queue server. Unlike in-memory servers such as [redis](, Darner is
+designed to handle queues much larger than what can be held in RAM. And unlike enterprise queue servers such as
+[RabbitMQ](, Darner keeps all messages **out of process**, relying instead on the kernel's
+virtual memory manager via [log-structured storage](
-A single darner server has a set of queues identified by name, which is also the filename of that queue's journal file.
-Each queue is a strictly-ordered FIFO of items of binary data.
+The result is a durable queue server that uses a small amount of in-resident memory regardless of queue size, while
+still achieving [remarkable performance](/wavii/darner/blob/master/docs/
-### Protocol
+Darner is based on Robey Pointer's [Kestrel](/robey/kestrel) simple, distributed message queue. Like Kestrel, Darner
+follows the "No talking! Shhh!" approach to distributed queues: A single Darner server has a set of queues identified
+by name. Each queue is a strictly-ordered FIFO, and querying from a fleet of Darner servers provides a loosely-ordered
+queue. Darner also supports Kestrel's two-phase reliable fetch: if a client disconnects before confirming it handled
+a message, the message will be handed to the next client.
-The official memcache protocol is described here:
+Compared to Kestrel, Darner boasts much higher throughput, better concurrency, an order of magnitude better tp99, and
+uses an order of magnitude less memory. But Darner has less configuration, and far fewer features than Kestrel. Check
+out the [benchmarks](/wavii/darner/blob/master/docs/!
-The darner implementation of the memcache protocol commands is described below.
+Darner is used at [Wavii](, and is written and maintained by [Erik Frey](/erikfrey).
-- `SET <queue-name> <# bytes>`
+## Installing
- Add an item to a queue. It may fail if the queue has a size or item limit and it's full.
+You'll need build tools, [CMake](, [Boost](, and
+[LevelDB]([snappy]( to build Darner:
-- `GET <queue-name>[options]`
+sudo apt-get install -y build-essential cmake libboost-all-dev libsnappy-dev
+tar xvzf leveldb-1.5.0.tar.gz && cd leveldb
+sudo mv libleveldb.* /usr/local/lib/ && sudo chown root:root /usr/local/lib/libleveldb.*
+cd ..
- Remove an item from a queue. It will return an empty response immediately if the queue is empty. The queue name may be
- followed by options separated by `/`:
+Then you can fetch and install Darner:
- - `/t=<milliseconds>`
+git clone
+cd darner
+cmake . && make && sudo make install
- Wait up to a given time limit for a new item to arrive. If an item arrives on the queue within this timeout, it's
- returned as normal. Otherwise, after that timeout, an empty response is returned.
+## Running
- - `/open`
+Make a directory for Darner to store its queues, say `/var/spool/darner/`, then run Darner like so.
- Tentatively remove an item from the queue. The item is returned as usual but is also set aside in case the client
- disappears before sending a "close" request. (See "Reliable Reads" below.)
+vagrant@ubuntu-oneiric:~/workspace/darner$ ./darner -d /var/spool/darner/
+[INFO] 2012-Aug-13 03:59:41.047739: darner: queue server
+[INFO] 2012-Aug-13 03:59:41.048051: build: Aug 12 2012 (22:24:28) v0.0.1 (c) Wavii, Inc.
+[INFO] 2012-Aug-13 03:59:41.048132: listening on port: 22133
+[INFO] 2012-Aug-13 03:59:41.048507: data dir: /var/spool/darner/
+[INFO] 2012-Aug-13 03:59:41.048798: starting up
- - `/close`
+Voila! By default, Darner listens on port 22133.
- Close any existing open read. (See "Reliable Reads" below.)
+## Protocol
- - `/abort`
+Darner follows the same protocol as [Kestrel](/robey/kestrel/blob/master/docs/, which is the memcache
- Cancel any existing open read, returing that item to the head of the queue. It will be the next item fetched.
- For example, to open a new read, waiting up to 500msec for an item:
- GET work/t=500/open
- Or to close an existing read and open a new one:
- GET work/close/open
-- `DELETE <queue-name>`
- Drop a queue, discarding any items in it, and deleting any associated journal files.
-- `FLUSH <queue-name>`
- Discard all items remaining in this queue. The queue remains live and new items can be added. The time it takes to
- flush will be linear to the current queue size, and any other activity on this queue will block while it's being
- flushed.
- Discard all items remaining in all queues. The queues are flushed one at a time, as if kestrel received a `FLUSH`
- command for each queue.
- Display the kestrel version in a way compatible with memcache.
-- `STATS`
- Display server stats in memcache style. They're described below.
+Currently missing from the Darner implementation but TODO: `/peek`, `FLUSH`, `FLUSH_ALL`, `DELETE`, and some stats.
Please sign in to comment.
Something went wrong with that request. Please try again.