node-kafka-streams

npm install --save kafka-streams

const {KafkaStreams} = require("kafka-streams");

const config = require("./config.json");
const factory = new KafkaStreams(config);

const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(..);

kstream.merge(ktable).filter(..).map(..).reduce(..).to("output-topic");

API Overview

• Quick Start
• API Info
• Documentation
• Operator descriptions
• Examples

README Overview

• Prerequisites
• Aim of this Library
• Description
• Port Progress Overview
• Operator Implementations
• Additional Operators
• Stream Action Implementations
• Join Operators Status
• Window Operations
• FAQ - More

Prerequisites

• kafka broker should be version >= 0.9.x
• nodejs should be version >= 6.10

Aim of this Library

• this is not a 1:1 port of the official JAVA kafka-streams
• the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
• stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example

Description

kafka-streams 🐙 equivalent for nodejs ✨🐢🚀✨ build on super fast 🔥 observables using most.js 🤘

ships with sinek 🙏 for backpressure

the lib also comes with a few window operations that are more similar to Apache Flink, yet they still feel natural in this api :squirrel:

overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

async (Promises) and sync stream operators e.g. stream$.map() or stream$.asyncMap()

super easy API

the lib is based on sinek, which is based on kafka-node's ConsumerGroups therefore it still requires a zookeeper connection (dont worry, your offset will be stored in the kafka broker)

Port Progress Overview

• core structure
• KStream base - stream as a changelog
• KTable base - stream as a database
• KStream & KTable cloning
• complex stream join structure
• advanced joins see
• windows (for joins) see
• flink like window operations
• word-count example
• more examples
• local-storage for etl actions
• local-storage factory (one per action)
• KStorage example for any DB that supports atomic actions
• backing-up local-storage via kafka
• kafka client implementation
• KTable replay to Kafka (produce)
• stream for topic message production only
• sinek implementation
• backpressure mode for KafkaClient
• auto-json payloads (read-map/write-map)
• auto producer partition and keyed-message handling
• documentation
• API description
• higher join & combine examples

Operator Implementations

• map
• asyncMap
• constant
• scan
• timestamp
• tap
• filter
• skipRepeats
• skipRepeatsWith
• slice
• take
• skip
• takeWhile
• skipWhile
• until
• since
• reduce
• chainReduce
• forEach (observe)
• chainForEach
• drain
• _zip
• _merge
• _join
• _combine
• _sample
• throttle
• debounce
• delay
• multicast
• A description of the operators can be found here
• Missing an operator? Feel free to open an issue 👮

Additional Operators

• mapStringToArray
• mapArrayToKV
• mapStringToKV
• mapParse
• mapStringify
• atThroughput
• mapWrapKafkaPayload
• mapToFormat
• mapFromFormat
• Want more? Feel free to open an issue 👮

Stream Action Implementations

• countByKey
• sumByKey
• min
• max
• Want more? Feel free to open an issue 👮

Join Operations

Operation description

KStream Status

• merge
• outerJoin
• innerJoin
• leftJoin

KTable Status

• merge
• outerJoin
• innerJoin
• leftJoin

KTable <-> KStream Status

• merge
• outerJoin
• innerJoin
• leftJoin

Window Operations

KStream

• window

More

Can I use this library yet?

Yes.

Are we ready for production yet?

No, please have some more patience 😄

Even More

Forks or Stars give motivation