Kafka SSE Example
This example project reads messages from a Kafka service and exposes the data over an HTTP streaming API using Server-Sent Events (SSE) protocol. It is written using Python & Django, and relies on Pushpin for managing the streaming connections.
Setup and use
Setup virtualenv and install dependencies:
virtualenv --python=python3 venv . venv/bin/activate pip install -r requirements.txt
Create a suitable
.env with Kafka and Pushpin settings:
Run the Django server:
python manage.py runserver
pushpin --route="* localhost:8000"
python manage.py relay
relay command sets up a Kafka consumer according to
KAFKA_CONSUMER_CONFIG, subscribes to all topics, and re-publishes received messages to Pushpin, wrapped in SSE format.
Clients can listen to events by making a request (through Pushpin) to
curl -i http://localhost:7999/events/test/
The output stream might look like this:
HTTP/1.1 200 OK Content-Type: text/event-stream Transfer-Encoding: chunked Connection: Transfer-Encoding event: message data: hello event: message data: world
How it works
This is a simple example project. In a real project you'd want to consider:
- Authenticating clients
- Only subscribing to Kafka topics for which there are listening clients