Thrash

Framework for data pipeline and streaming in Javascript

WIP this is not ready to be used.

Install

  npm install --save thrash

Usage

API Usage

Using the high level Framework with async data pipeline and dependencies tracking

  import Thrash, {kafka, winston} from 'thrash';
  const cfg = { ... };

  Thrash.init({
    receiver: {
      input: kafka.consumer(cfg, cfg.topics)
      process: (obs) => obs.map((v) => JSON.parse(v.value)).share()
    },
    stat: {
      input: ['receiver'],
      process: (obs) => obs
        .bufferTime(1000)
        .map((values) => `${values.length}/s`)
        .subscribe()
    },
    log: {
      input: ['receiver'],
      process: (obs) =>
        obs.subscribe({
          next: (msg) => console.log('received', msg)
        })
    }
  });

Using the data-sources directly

  import { kafka, websocket } from 'thrash';

  const cfg = {
    kafka: {
      topic: { topic: 'test', partition: 0 },
      host: '127.0.0.1',
      port: 2181,
      options: {}
    },
    ws: {
      url: 'ws://127.0.0.1:1337',
      send: {
        mask: true
      }
    }
  };

  kafka
    .consumer(cfg.kafka, [cfg.kafka.topic])
    .map((v) => v.value)
    .bufferCount(1024)
    .subscribe(websocket.producer(cfg.ws));