Find file
Fetching contributors…
Cannot retrieve contributors at this time
65 lines (48 sloc) 1.58 KB


Batching your streams since 2012.


Your logging service takes a stream of JSON objects. You don't want to stream objects as they come, but rather batch them and send them every minute, or when a hundred comes by or some such thing.

var Batcher = require('batcher');

var b = new Batcher({
  batchSize: 100,
  batchTimeMs: 60000,
  encoder: function(items) {

b.pipe(net.createConnection(host: '', port: 1337));

More frequently, you don't need to pipe it but just accumulate:

b.on('data', function(payload) {
  upload('/somewhere', payload);

You can also manually flush the batch:

b.flush(function() {
  console.log('one has been sent');

Batchers support write(), pause(), resume(), flush(), end() and destroy().

pause()/resume() is useful when you need to send with retries, but want to avoid the thundering herd when the server comes back online.

var b = new Batcher({
  batchSize: 1,
  maxBatchSize: 100,
  encoder: function(items) {

    function send() {
        url: some_url,
        json: items,
      }, function(err) {
        if (err) return setTimeout(send, 1000);


batchSize of 1 means send them as they come. maxBatchSize of 100 means if there are more available, send as many as 100. setTimeout is used to retry sending every second.