service-broker-kafka

Small proof of concept on how to route log messages from one kafkaConsumerApp to a log consumer through a kafka broker

How kafka works

See this

how to run the broker

• install docker, docker-compose

cd service-broker-kafka
docker compose -f docker-compose-dev.yml up

Install intellij kafka plugin or any other kafka client

Connect on port 9094

how to run the service-producer-1

• install java-jdk-17, some IDE (intellij is better)

cd service-producer-1
sh ./gradlew build ; sh ./gradlew bootRun

First producer uses default logback logging spring boot subsystem and does not work yet. Messages to kafka streams must be sent in a explicit way

@Service
class SimpleJob(val template: KafkaTemplate<String?, String?>) {

    private val logger by lazy { LoggerFactory.getLogger(SimpleJob::class.java) }

    @Scheduled(fixedRate = 5000)
    fun work() {
        val date = "${Date()}"
        logger.info(date)
        template.send("logs", date)
    }
}

how to run the service-producer-2

• install java-jdk-17, some IDE (intellij is better)

cd service-producer-2
sh ./gradlew build ; sh ./gradlew bootRun

Second producer uses log4j2 logging spring boot subsystem and logs messages to a kafka topic in a transparent way. Just log your message and go home.

There is an issue where it starts to log warnings generated by kafka producer itself, and then it enters into an infinite loop. I am working on it.

how to run the service-consumer-1

• install java-jdk-17, some IDE (intellij is better)

cd service-consumer-1
sh ./mvnw compile ; sh ./mvnw exec:java

This project aims to use as little boilerplate as possible. it uses javalin, a lightweight java/kotlin web framework and kafka-client libraries directly.

Kafka consumers subscribes to topics and then starts to poll them in real time, synchronously.

That means you're not supposed to consume kafka streams in the main thread of a web server, it will hang.

Therefore, this project spawns another thread and then polls kafka messages from there.

how to run service-consumer-2

This project uses spring-boot-kafka to consume the streams.

cd service-consumer-2
sh ./mvnw compile ; sh ./mvnw spring-boot:run