Expose Kafka messages via HTTP streaming API
Switch branches/tags
Nothing to show
Clone or download
Latest commit 67848fe Jun 25, 2018
Type Name Latest commit message Commit time
Failed to load latest commit information.


Kafka SSE Example

This example project reads messages from a Kafka service and exposes the data over an HTTP streaming API using Server-Sent Events (SSE) protocol. It is written using Python & Django, and relies on Pushpin for managing the streaming connections.

Setup and use

Setup virtualenv and install dependencies:

virtualenv --python=python3 venv
. venv/bin/activate
pip install -r requirements.txt

Create a suitable .env with Kafka and Pushpin settings:


Run the Django server:

python manage.py runserver

Run Pushpin:

pushpin --route="* localhost:8000"

Run the relay command:

python manage.py relay

The relay command sets up a Kafka consumer according to KAFKA_CONSUMER_CONFIG, subscribes to all topics, and re-publishes received messages to Pushpin, wrapped in SSE format.

Clients can listen to events by making a request (through Pushpin) to /events/{topic}/:

curl -i http://localhost:7999/events/test/

The output stream might look like this:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Transfer-Encoding: chunked
Connection: Transfer-Encoding

event: message
data: hello

event: message
data: world

How it works

The code is relatively simple. views.py sets up the SSE endpoint and relay.py does the message I/O.


This is a simple example project. In a real project you'd want to consider:

  • Authenticating clients
  • Only subscribing to Kafka topics for which there are listening clients