An implementation of Amazon's Kinesis built on LevelDB
Latest commit c8685eb Jun 19, 2018

README.md

Kinesalite

Build Status

An implementation of Amazon's Kinesis, focussed1 on correctness and performance, and built on LevelDB (well, @rvagg's awesome LevelUP to be precise).

The Kinesis equivalent of dynalite.

To read and write from Kinesis streams in Node.js, consider using the kinesis module.

NOTE: Starting with v1.0, SSL/HTTPS is no longer the default. Use --ssl to enable pre-v1.0 behaviour.

Example

$ kinesalite --help

Usage: kinesalite [--port <port>] [--path <path>] [--ssl] [options]

A Kinesis http server, optionally backed by LevelDB

Options:
--help                 Display this help message and exit
--port <port>          The port to listen on (default: 4567)
--path <path>          The path to use for the LevelDB store (in-memory by default)
--ssl                  Enable SSL for the web server (default: false)
--createStreamMs <ms>  Amount of time streams stay in CREATING state (default: 500)
--deleteStreamMs <ms>  Amount of time streams stay in DELETING state (default: 500)
--updateStreamMs <ms>  Amount of time streams stay in UPDATING state (default: 500)
--shardLimit <limit>   Shard limit for error reporting (default: 10)

Report bugs at github.com/mhart/kinesalite/issues

Or programmatically:

// Returns a standard Node.js HTTP server
var kinesalite = require('kinesalite'),
    kinesaliteServer = kinesalite({path: './mydb', createStreamMs: 50})

// Listen on port 4567
kinesaliteServer.listen(4567, function(err) {
  if (err) throw err
  console.log('Kinesalite started on port 4567')
})

Once running, here's how you use the AWS SDK to connect (after configuring the SDK):

var AWS = require('aws-sdk')

var kinesis = new AWS.Kinesis({endpoint: 'http://localhost:4567'})

kinesis.listStreams(console.log.bind(console))

Or with the kinesis module (currently only works in https mode, when kinesalite is started with --ssl):

var kinesis = require('kinesis')

kinesis.listStreams({host: 'localhost', port: 4567}, console.log)

CBOR protocol issues with the Java SDK

The Java AWS SDK recently changed their Kinesis client to default to the CBOR protocol, which kinesalite doesn't support – you may see an error like this:

com.amazonaws.AmazonServiceException: Unable to parse HTTP response content (Service: AmazonKinesis; Status Code: 404; Error Code: null;

You can set the AWS_CBOR_DISABLE environment variable to disable this (any value should work, eg true or 1) before invoking any of the Kinesis calls in the Java SDK.

Installation

With npm do:

$ npm install -g kinesalite

Footnotes

1Hi! You're probably American (and not a New Yorker editor) if you're worried about this spelling. No worries – and no need to open a pull request – we have different spellings in the rest of the English speaking world 🐨