Skip to content

A Kafka Streams reference application and framework playground

Notifications You must be signed in to change notification settings

astubbs/ks-tributary

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

  • Badges

Conventional Commits

  • Build

Drone:

Build Status

CodeCov.io:

codecov

KS-Tributary; A Kafka Streams Reference App

Realistic implementation of a full real Kafka Streams application and stream processing logic.

How to run

Integration tests require a running docker daemon.

Intellij - use provided run targets to run with or without integration tests

Maven

Disable integration tests

mvn clean verify -DskipITs

How to build

Features

Stream Processing

Generic

  • Realtime delta calculation (e.g. produce derived delta messages from database CDC full state messages)
  • Cached system configuration lookup with realtime updates
  • Realtime fan out of topic data onto communication channels like web sockets
  • Dynamic Avro wrapper system to extend Avro objects with custom code without modifying generated code, integrated with Avro Serdes

Finance

  • Realtime calculation of windows
  • Three topologies
    • Financial snap set precomputation
    • Simpler approach that's centralised performance wise
      • More sophisticated fully distributed
      • Custom state store intermediate level demonstration
  • Simple high low bar calculation
  • Back loading of data from previous buckets

Chat

  • Transcript generation

Unit testing

  • Demonstration of Time model decoupling for test
  • Demonstration of AssertJ powerful assertions
    • here
    • here

Integration testing

  • Model hydration using human readable data from JavaFaker[] project
    • Deterministic generation of series of random data, useful for larger integration tests
  • TestContainers for Integration Testing with real Kafka brokers.
  • CI service integration: Travis, CircleCI, Drone (drone not working with Docker in Docker (i.e. TestContainers not working))
  • JSR Money integration
  • Avro message sizing analysis - i.e. how big is my data when it's bin packed and compressed? Compared with Gzipped vs Zstd vs Json?
  • Demonstration of asynchronous testing using Awaitility[]

Software Structure

  • Guice (dynamic) and Dagger (static) dependency injection examples
  • Dagger
  • Java Money library demonstration
  • Java Units of Measure library demonstration (statically typed Scientific units combination library)
  • Lombok use (make Java more tolerable)
  • Avro IDL vs AVSC model generation demonstration

Features on their way...

Stream Processing

  • Realtime out of order recalculation of data aggregation in state dependent buckets
  • Massive scale question and answer quiz show of just in time answer discovery
    • Massive skew avoidance techniques
  • Compound key query demonstration / data demoralisation (e.g. Invoice reconstruction from Orders and Order Items)
  • Yearly aggregator by date (think April 5th Financial years)
    • Normal window computations can only run against fixed size windows (e.g. not calendar dates)
    • Extending to also do monthly dates is an easy modification

Software Structure

  • External Interactive Query example on Qurkus[] and GraalVM[]

[0.7] Dagger

[0.6] 0.5 0.4 0.3 0.2 [0.1]

  • Three topologies
    • Financial snap set precomputation

    • Simpler approach that's centralised performance wise

      • More sophisticated fully distributed
        • Custom state store intermediate level demonstration
  • Unit testing coverage
    • Guice test wiring
  • Separated Integration testing
    • Kafka ContainerTests
  • Avro message model sizing analysis
    • Raw bin-packed vs GZipped vs Json
  • Java Science Units and Quantities use

Development

Building

  • Requires Lombok plugin for your IDE

Contributing

Submit a PR for review :)

https://www.conventionalcommits.org/en/v1.0.0/ the Angular convention

About

A Kafka Streams reference application and framework playground

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages