CouchDB as the stream/commit-log

[twilson63]

CouchDB _changes

One of the most powerful features of couchdb is its ability to notify any application when changes have occurred in a database. This ability makes it super simple to get notified when documents are created or modified.

There are several ways to connect to the changes feed, you can use polling, long polling, or continuous.

curl -X GET $HOST/db/_changes?since=1
curl -X GET "$HOST/db/_changes?feed=longpoll&since=2"
curl -X GET "$HOST/db/_changes?feed=continuous&since=3"

Continuous feeds are great for applications other than browsers, long polling is the current best option for browsers, or you could combine continuous and web-sockets as well.

Each time a change event occurs a change message is emitted and on the change message is a sequence id, this id is the commit-log id, and you can store this id so if your service is interrupted in any way, it can start listening where it left off. One of the challenges is finding the right place to store that information.

CouchDB _local is a feature that allows you to read and write to local documents in the couchdb store, these documents do not show up in the changes log and they do not keep large revision histories, so that they can handle heavy updates without suffering from large growth in your couchdb.

You can add a document to the _local/ section like so:

curl -X PUT $HOST/db/_local/<mydoc> -d '{"FOO": "BAR"}'

In the Palmetto concept we talk about a stream or commit-log, CouchDb is a great choice for this piece of the palmetto flow. It provides a changes-feed and is an append-only structured document store, you can leverage couchdb to be heart of your application, with cors support you can isolate your web client apps that contain your components and your service container apps that contain your services as separate isolated processes to your stream/commit-log.

Below is an example, that uses the changes feed to listen for new event messages in the stream commit-log:

var EventEmitter = require('events').EventEmitter;
var ee = new EventEmitter();

module.exports = function(commitlog, svc) {
  var config = '_local/' + svc;
  // need to consume commit log and notify services
  commitlog.get(config).then(function(doc) {
    return doc.seq;
  }, function(err) {
    commitlog.put({ _id: config, seq: 'now' })
    return 'now';
  })
  .then(function(seq) {
    commitlog.changes({
      since: seq,
      live: true,
      include_docs: true
    }).on('change', function(change) { 
      var doc = change.doc;
      if (doc.model && doc.verb) {
        var topic = [change.doc.model, change.doc.verb].join('-');
        ee.emit(topic, change.doc);      
      }
      commitlog.get(config).then(function(d) {
        d.seq = change.seq;
        commitlog.put(d);
      });
    });
  })
  .catch(function(err) {
    console.log(err);
  });

  // listen for responses and post to commit log
  ee.on('post', function(event) {
    var topic = [event.model, event.verb].join('-');
    topic = [topic, (new Date()).toISOString()].join('\x00');  
    commitlog.put(event, topic)
  });

  return ee;
}; 

The goal of this module is to be able to drop it into any service container and then register your services and your done, you have a fully functional services container! This module could also be used to message components by leveraging web-sockets or couchdb/pouchdb replication!