Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 

scala-kafka-protobuf

A PoC that defines single-message topic schemas for Apache Kafka using Protocol Buffers (Protobuf) and Scala.

The goal for this PoC is to evaluate the usage of a single envelope protobuf definition per kafka topic. The main reason behind that is having a clear topic schema and clear expectations for consumers of these messages.

I've written a blog post based on this PoC which explains this concept in a bit more details. The blog post can be found here

P.S. Feel also free to use this PoC as a quick start project using Scala, Kafka and Protobuf.

Description

The idea is to have an envelope per topic defined in protobuf. We can add any header fields we need (e.g. correlation id) but the payload must use protobuf's oneof to list all the message types are sent on that topic.

Example

In the users.proto file we have got a basic protobuf definitions for the users topic:

// protobuf file headers ..

message UsersEnvelope {
    string correlation_id = 1;
    // some other metadata
    oneof payload {
        UserCreated user_created = 11;
        UserUpdated user_updated = 12;
        UserActivated user_activated = 13;
    }
}

// the rest of the messages ..

Each protobuf compiler will generate these classes in its own way along with an enum or some class hierarchy for the oneof cases (depending on the language). Usually it will also generate a special Empty case (in Scala, Java and C# at least, the name varies per language) to handle invalid or missing messages (cases).

Tech stack

Running

This already assumes you have clonned this repository and installed sbt and docker.

  • First, we need to start Kafka. This repository has a docker-compose file provided. All we need to do is to run:
docker-compose -f docker/docker-compose.yml up -d 
  • To start our app:
sbt "runMain com.tudorzgureanu.Main"  
  • Now we are ready to send some messages to kafka. For that we would use our KafkaMessageSender utility class. Feel free to change this class locally to send any messages you prefer. Finally, simply run:
sbt "runMain com.tudorzgureanu.KafkaMessageSender"

About

PoC using scala that defines single-message protobuf schema per Kafka topic

Topics

Resources

Releases

No releases published

Packages

No packages published

Languages