Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 

README.md

Kafka SSE Example

This example project reads messages from a Kafka service and exposes the data over an HTTP streaming API using Server-Sent Events (SSE) protocol. It is written using Python & Django, and relies on Pushpin for managing the streaming connections.

Setup and use

Setup virtualenv and install dependencies:

virtualenv --python=python3 venv
. venv/bin/activate
pip install -r requirements.txt

Create a suitable .env with Kafka and Pushpin settings:

KAFKA_CONSUMER_CONFIG={"bootstrap.servers":"localhost:9092","group.id":"mygroup"}
GRIP_URL=http://localhost:5561

Run the Django server:

python manage.py runserver

Run Pushpin:

pushpin --route="* localhost:8000"

Run the relay command:

python manage.py relay

The relay command sets up a Kafka consumer according to KAFKA_CONSUMER_CONFIG, subscribes to all topics, and re-publishes received messages to Pushpin, wrapped in SSE format.

Clients can listen to events by making a request (through Pushpin) to /events/{topic}/:

curl -i http://localhost:7999/events/test/

The output stream might look like this:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Transfer-Encoding: chunked
Connection: Transfer-Encoding

event: message
data: hello

event: message
data: world

How it works

The code is relatively simple. views.py sets up the SSE endpoint and relay.py does the message I/O.

Limitations

This is a simple example project. In a real project you'd want to consider:

  • Authenticating clients
  • Only subscribing to Kafka topics for which there are listening clients

About

Expose Kafka messages via HTTP streaming API

Resources

Releases

No releases published

Packages

No packages published

Languages