Skip to content
Telstar makes it easy to write consumer groups and producers against redis streams
Python Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci
telstar Give the user a neater way to deal with errors (#12) Jul 25, 2019
.gitignore
LICENSE
README.md
pyproject.toml
requirements.txt
test_telstar.py

README.md

Project logo

Telstar

Status CircleCI Pypi GitHub Issues GitHub Pull Requests License


This library is what came out of creating a distributed service architecture for one of our products. Telstar makes it easy to write consumer groups and producers against redis streams.

📝 Table of Contents

🧐 About

In order to run our distributed architecture at @bitspark we needed a way to receive and produces messages with an exactly once delivery design. We think that, by packing up our assumptions into a separate library we can make it easy for other services to adhere to our initial design and/or comply to future changes.

Another aspect for open sourcing this library is that we have not found a package that does what we needed. And as we ourselves use OSS in many critical part of our infrastructure we wanted to give back this project.

🏁 Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites

You will need python >= 3.6 as we use type annotations and a running redis server with at least version >= 5.0.

Installing

A step by step series of examples that tell you how to get a development env running. Install everything using pip

pip install telstar

🔧 Running the tests

This package comes with an end to end test which simulates a scenario with all sorts of failures that can occur during operation. Here is how to run the tests.

git clone git@github.com:Bitspark/telstar.git
pip install -r requirements.txt

./telstar/tests/test_telstar.sh
pytest --ignore=telstar/

🎈 Usage

This package uses consumer groups and redis streams as a backend to deliver messages exactly once. In order to understand redis streams and what the Consumer can do for you to read go and read Redis Streams

The Producer - how to get data into the system

Create a producer in our case this produces different message with the same data every .5 seconds

import redis
import os
from uuid import uuid4
from time import sleep

from telstar.producer import Producer
from telstar.com import Message

link = redis.from_url("redis://")

def producer_fn():
    topic = "userSignedUp"
    uuid = uuid4() # Based on this value the system deduplicates messages
    data = dict(email="test1@example.com", field="value")
    msgs = [Message(topic, uuid, data)]

    def done():
        # Do something after all messages have been sent.
        sleep(.5)
    return msgs, done

Producer(link, producer_fn, context_callable=None).run()

Start the producer with the following command

python producer.py

The Consumer - how to get data out of the system

Now lets creates consumer

import redis
import telstar

from marshmallow import EXCLUDE, Schema, fields

redis = redis.from_url("redis://")
consumer_name = "local.dev"

app = telstar.app(redis, consumer_name=consumer_name, block=20 * 1000)


class UserSchema(Schema):
    email = fields.Email(required=True)

    class Meta:
        unknown = EXCLUDE


@app.consumer("mygroup", ["userSignedUp"], schema=UserSchema, strict=True, acknowledge_invalid=False)
def consumer(record: dict):
    print(record)

if __name__ == "__main__":
    app.start()

Now start the consumer as follows:

python consumer.py

You should see output like the following

{"email": "test1@example.com"}
{"email": "test1@example.com"}
{"email": "test1@example.com"}

Until the stream is exhausted.

🚀 Deployment

We currently use Kubernetes to deploy our producers and consumers as simple jobs, which of course is a bit suboptimal, it would be better to deploy them as replica set.

⛏️ Built Using

✍️ Authors

See also the list of contributors who participated in this project.

🎉 Acknowledgements

You can’t perform that action at this time.