EventStream_to_Kafka

Kafka Producer produces data from EventStream (SSE - Server Sent Events) and publish to the Topic and the Kafka Consumers can consume that messages

Using the Python the client library sseclient to iterate over the http Server Sent Event (SSE) streams (also known as EventSource, after the name of the Javascript interface inside browsers).

Installation

pip install sseclient

Demo

Start the Zookeeper and the Kafka Server

zookeeper-server-start.sh ~/Desktop/kafka_2.13-3.2.1/config/zookeeper.properties

kafka-server-start.sh ~/Desktop/kafka_2.13-3.2.1/config/server.properties

Producer

python3 kafka_producer.py gets the data from the Wikimedia EventStream and publish the message to the Kafka Topic wikimedia.recentchange

Consumer

python3 kafka_consumer.py wikimedia.recentchange cg1 consumes the message from the topic wikimedia.recentchange and the consumer is in the Consumer Group cg1.

The Producer will produce lots of data since it is reading from the EventSource.