Skip to content
Switch branches/tags

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Project logo


Status CircleCI Pypi GitHub Issues GitHub Pull Requests License

This library is what came out of creating a distributed service architecture for one of our products. Telstar makes it easy to write consumer groups and producers against Redis streams.

📝 Table of Contents

🧐 About

To run our distributed architecture at @bitspark we needed a way to produce and consume messages with an exactly-once delivery design. We think that, by packing up our assumptions into a separate library, we can make it easy for other services to adhere to our initial design and/or comply with future changes.

Another aspect of open-sourcing this library is that we have not found a package that does what we needed. And as we use OSS in many critical parts of our infrastructure, we wanted to give back this project.

🏁 Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.


You will need python >= 3.6 as we use type annotations and a running Redis server with at least version >= 5.0.


A step by step series of examples that tell you how to get a development env running. Install everything using pip.

pip install telstar

🔧 Running the tests

This package comes with an end to end test, which simulates a scenario with all sorts of failures that can occur during operation. Here is how to run the tests.

git clone
pip install -r requirements.txt

pytest --ignore=telstar/

🎈 Usage

This package uses consumer groups and Redis streams as a backend to deliver messages exactly once. To understand Redis streams and what the Consumer can do for you to read go and read Redis Streams

The Producer - how to get data into the system

Create a producer in our case this produces a different message with the same data every .5 seconds

import Redis
import os
from uuid import uuid4
from time import sleep

from telstar.producer import Producer
from import Message

link = redis.from_url("redis://")

def producer_fn():
    topic = "userSignedUp"
    uuid = uuid4() # Based on this value the system deduplicates messages
    data = dict(email="", field="value")
    msgs = [Message(topic, uuid, data)]

    def done():
        # Do something after all messages have been sent.
    return msgs, done

Producer(link, producer_fn, context_callable=None).run()

Start the producer with the following command.


The Consumer - how to get data out of the system

Now let's create consumer

import Redis
import telstar

from marshmallow import EXCLUDE, Schema, fields

redis = redis.from_url("redis://")
consumer_name = ""

app =, consumer_name=consumer_name, block=20 * 1000)

class UserSchema(Schema):
    email = fields.Email(required=True)

    class Meta:
        unknown = EXCLUDE

@app.consumer("mygroup", ["userSignedUp"], schema=UserSchema, strict=True, acknowledge_invalid=False)
def consumer(record: dict):

if __name__ == "__main__":

Now start the consumer as follows:


You should see output like the following.

{"email": ""}
{"email": ""}
{"email": ""}

Until the stream is exhausted.

🚀 Deployment

We currently use Kubernetes to deploy our producers and consumers as simple jobs, which, of course, is a bit suboptimal. It would be better to deploy them as a replica set.

⛏️ Built Using

✍️ Authors

See also the list of contributors who participated in this project.

🎉 Acknowledgements


Telstar makes it easy to write consumer groups and producers against redis streams








No releases published