Event sourcing journal implementation using Kafka as main storage
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
eventual-cassandra/src
io/src/main/scala/com/evolutiongaming/kafka/journal
journal-prometheus/src
journal/src
persistence-tests/src/test
persistence/src
project
replicator-prometheus/src
replicator/src
scripts
.gitignore
.travis.yml
LICENSE
README.md
build.sbt
notes.txt
version.sbt

README.md

Kafka Journal Build Status Coverage Status Codacy Badge version License: MIT

Stream data from two sources when one is eventually consistent and the other one loses own tail

This library provides ability to use kafka as storage for events. Kafka is a perfect fit in case you want to have streaming capabilities for your events However it also uses cassandra to keep data access performance on acceptable level and overcome kafka retention policy Cassandra is a default choice, but you may use any other storage which satisfies following interfaces:

High level idea

Writing events flow:

  1. Journal client publishes events to kafka
  2. Replicator app stores events to cassandra

Reading events flow:

  1. Client publishes special marker to kafka so we can make sure there are no more events to expect
  2. Client reads events from cassandra, however at this point we are not yet sure that all events are replicated from kafka to cassandra
  3. Client read events from kafka using offset of last event found in cassandra
  4. We consider recovery finished when marker found in kafka

Notes

  • Kafka topic may be used for many different entities
  • We don't need to store all events in kafka as long as they are in cassandra
  • We do not cover snapshots yet
  • Replicator is a separate application
  • It is easy to replace cassandra here with some relational database

State recovery performance

Reading events performance depends on finding the closest offset to the marker as well on replication latency (time difference between the moment event has been written to kafka and the moment when event gets into cassandra)

We may share same kafka consumer for many simultaneous recoveries

Read & write capabilities:

  • Client allowed to read + write kafka and read cassandra
  • Replicator allowed to read kafka and read + write cassandra

Hence we recommend to configure access rights accordingly.

Troubleshooting

Kafka exceptions in logs

Kafka client is tend to log some exceptions at error level, however in reality those are harmless in case of operation retried successfully Retriable exceptions usually extend RetriableException

Here is the list of known error logs you may ignore:

  • Offset commit failed on partition .. at offset ..: The request timed out.
  • Offset commit failed on partition .. at offset ..: The coordinator is loading and hence can't process requests.
  • Offset commit failed on partition .. at offset ..: This is not the correct coordinator.
  • Offset commit failed on partition .. at offset ..: This server does not host this topic-partition.

Akka persistence plugin

In order to use kafka-journal as akka persistence plugin you would need to add following to your *.conf file:

akka.persistence.journal.plugin = "evolutiongaming.kafka-journal.persistence.journal"

Unfortunately akka persistence snapshot plugin is not implemented yet.

Setup

resolvers += Resolver.bintrayRepo("evolutiongaming", "maven")

libraryDependencies += "com.evolutiongaming" %% "kafka-journal" % "0.0.10"

libraryDependencies += "com.evolutiongaming" %% "kafka-journal-persistence" % "0.0.10"

libraryDependencies += "com.evolutiongaming" %% "kafka-journal-replicator" % "0.0.10"

libraryDependencies += "com.evolutiongaming" %% "kafka-journal-eventual-cassandra" % "0.0.10"

libraryDependencies += "com.evolutiongaming" %% "kafka-journal-prometheus" % "0.0.10"

libraryDependencies += "com.evolutiongaming" %% "kafka-journal-replicator-prometheus" % "0.0.10"