consistent hash ring of leveldbs
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


A leveldb-backed consistent hash ring, for your caching needs. I can see the day when you might want to put data in this & feel reasonably sort of confident you might get it back out again. If this scares you, it's supposed to.

on npm Tests Coverage Status Dependencies

One part each

Rod Vagg's levelup leveldb bindings for node + sublevel to create buckets.

Dominic Tarr's crtd, which uses his scuttlebutt implementation to keep a document in sync.

light-cycle, a lightweight consistent hash ring structure that can be mixed into most anything.

restify to provide a simple RESTful api to data in the buckets.

Shake with ice

Run a server:

node index.js --id=node-one -m -p 3333 -g 4114 -d ./db | ./node_modules/.bin/bunyan -o short

Run a client or five:

node index.js --id=node-two -p 3334 -g 4114 -s -d ./db2 | ./node_modules/.bin/bunyan -o short
node index.js --id=node-three -p 3335 -g 4114 -s -d ./db3 | ./node_modules/.bin/bunyan -o short
node index.js --id=node-four -p 3336 -g 4114 -s -d ./db4 | ./node_modules/.bin/bunyan -o short

Replace with the IP address of your server.

Strain into a chilled glass

Then stuff some data in:

http PUT name="Sobieski" rating=5
http PUT name="Tito's Handmade" rating=5
http PUT name="Bimber" rating=4

Get it back out: http GET | json

(Human-friendly shell commands courtesy of httpie).

Garnish with a twist of orange peel

API endpoints exposed:

  • GET /:bucket: stream sorted keys for a bucket (works!)
  • POST /:bucket: add an item to the cache; id is generated for you & returned
  • PUT /:bucket/:id: add/update an item in the cache; 204 response
  • GET /:bucket/:id: get an item from the cache
  • HEAD /:bucket/:id: headers for an item
  • DEL /:bucket/:id: remove an item


Send this header to specify a time-to-live for your cached data: X-Scurry-TTL: [seconds]

Conditional requests

Scurry sends an ETag header and a last-modified timestamp.

Storage format

As of version 0.0.4, the data stored in the LevelDB nodes is json structured as follows:

	version:        1,                       // storage version
    ts:   ,              // timestamp of last set()
    payload:        value,                   // base64-encoded string if buffer, JSON string if not
    etag:           crc.digest('hex'),       // md5 hex digest of payload
    base64:         valueIsB64String,        // true if the payload had to be base64 encoded
    'content-type': metadata['content-type'] // content-type if passed in


Upcoming releases:

  • Release 0.0.4 will probably make streaming keys work, maybe. Done!
  • Release 0.0.5 will finalize the storage format. Probably done!
  • Release 0.0.6 will contemplate eviction. Not yet!

General goals:

  • Implement key streaming from multiple nodes. See notes in endpoints.handleGetBucket().
  • The RESTful server needs error handling.
  • Reconnect on errors.
  • Error handling.
  • Better logging. Configurable, for one thing.
  • Back ends should be pluggable; the API is very small.
  • Stretch goal: replication?