PoC using scala that defines single-message protobuf API per Kafka topic
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docker
project
src/main
.gitignore
.scalafmt.conf
README.md
build.sbt

README.md

scala-kafka-protobuf

A PoC that defines single-message topic schemas for Apache Kafka using Protocol Buffers (Protobuf) and Scala.

The goal for this PoC is to evaluate the usage of a single envelope protobuf definition per kafka topic. The main reason behind that is having a clear topic schema and clear expectations for consumers of these messages.

I've written a blog post based on this PoC which explains this concept in a bit more details. The blog post can be found here

P.S. Feel also free to use this PoC as a quick start project using Scala, Kafka and Protobuf.

Description

The idea is to have an envelope per topic defined in protobuf. We can add any header fields we need (e.g. correlation id) but the payload must use protobuf's oneof to list all the message types are sent on that topic.

Example

In the users.proto file we have got a basic protobuf definitions for the users topic:

// protobuf file headers ..

message UsersEnvelope {
    string correlation_id = 1;
    // some other metadata
    oneof payload {
        UserCreated user_created = 11;
        UserUpdated user_updated = 12;
        UserActivated user_activated = 13;
    }
}

// the rest of the messages ..

Each protobuf compiler will generate these classes in its own way along with an enum or some class hierarchy for the oneof cases (depending on the language). Usually it will also generate a special Empty case (in Scala, Java and C# at least, the name varies per language) to handle invalid or missing messages (cases).

Tech stack

Running

This already assumes you have clonned this repository and installed sbt and docker.

  • First, we need to start Kafka. This repository has a docker-compose file provided. All we need to do is to run:
docker-compose -f docker/docker-compose.yml up -d 
  • To start our app:
sbt "runMain com.tudorzgureanu.Main"  
  • Now we are ready to send some messages to kafka. For that we would use our KafkaMessageSender utility class. Feel free to change this class locally to send any messages you prefer. Finally, simply run:
sbt "runMain com.tudorzgureanu.KafkaMessageSender"