Skip to content
Stream Framework is a Python library, which allows you to build news feed, activity streams and notification systems using Cassandra and/or Redis. The authors of Stream-Framework also provide a cloud service for feed technology:
Branch: master
Clone or download

Latest commit

tbarbugli Merge pull request #236 from timgates42/bugfix/typo_whether
Fix simple typo: wether -> whether
Latest commit e8ab8e9 Jan 6, 2020


Type Name Latest commit message Commit time
Failed to load latest commit information.
docs Fix simple typo: wether -> whether Jan 4, 2020
stream_framework Replace async by async_ Dec 24, 2019
.coveragerc more renaming Sep 23, 2014
.coveralls.yml cleanup Jul 19, 2013
.gitignore Add some files to gitignore Oct 18, 2016
.travis.yml Test against newer Cassandra driver Oct 28, 2016
AUTHORS Add myself to the contributors Oct 17, 2016
CHANGES release 1.4.0 Oct 18, 2016
LICENSE.txt feedly first commit Aug 8, 2012 pip shouldnt include the example app Oct 15, 2013 small fixes Aug 10, 2019
ROADMAP Update ROADMAP Mar 26, 2015
Vagrantfile fix inline provisioner for vagrant Sep 11, 2013 more renaming work Sep 23, 2014
pytest.ini more renaming work Sep 23, 2014 Use extras_require Oct 17, 2016

Stream Framework

Build Status StackShare

Activity Streams & Newsfeeds

Examples of what you can build

Stream Framework is a Python library which allows you to build activity streams & newsfeeds using Cassandra and/or Redis. If you're not using python have a look at Stream, which supports Node, Ruby, PHP, Python, Go, Scala, Java and REST.

Examples of what you can build are:

  • Activity streams such as seen on Github
  • A Twitter style newsfeed
  • A feed like Instagram/ Pinterest
  • Facebook style newsfeeds
  • A notification system

(Feeds are also commonly called: Activity Streams, activity feeds, news streams.)


Build scalable newsfeeds and activity streams using

Stream Framework's authors also offer a web service for building scalable newsfeeds & activity streams at Stream. It allows you to create your feeds by talking to a beautiful and easy to use REST API. There are clients available for Node, Ruby, PHP, Python, Go, Scala and Java. The Get Started page explains the API & concept in a few clicks. It's a lot easier to use, free up to 3 million feed updates and saves you the hassle of maintaining Cassandra, Redis, Faye, RabbitMQ and Celery workers.

Background Articles

A lot has been written about the best approaches to building feed based systems. Here's a collection of some of the talks:

Stream Framework


Installation through pip is recommended::

$ pip install stream-framework

By default stream-framework does not install the required dependencies for redis and cassandra:

Install stream-framework with Redis dependencies

$ pip install stream-framework[redis]

Install stream-framework with Cassandra dependencies

$ pip install stream-framework[cassandra]

Install stream-framework with both Redis and Cassandra dependencies

$ pip install stream-framework[redis,cassandra]

Authors & Contributors


Example application

We've included a Pinterest-like example application based on Stream Framework.


Using Stream Framework

This quick example will show you how to publish a "Pin" to all your followers. So let's create an activity for the item you just pinned.

from stream_framework.activity import Activity

def create_activity(pin):
    activity = Activity(
        time=make_naive(pin.created_at, pytz.utc),
    return activity

Next up we want to start publishing this activity on several feeds. First of all, we want to insert it into your personal feed, and then into your followers' feeds. Let's start by defining these feeds.

from stream_framework.feeds.redis import RedisFeed

class UserPinFeed(PinFeed):
    key_format = 'feed:user:%(user_id)s'

class PinFeed(RedisFeed):
    key_format = 'feed:normal:%(user_id)s'

Writing to these feeds is very simple. For instance to write to the feed of user 13 one would do:

feed = UserPinFeed(13)

But we don't want to publish to just one users feed. We want to publish to the feeds of all users which follow you. This action is called a "fanout" and is abstracted away in the manager class. We need to subclass the Manager class and tell it how we can figure out which users follow us.

from stream_framework.feed_managers.base import Manager

class PinManager(Manager):
    feed_classes = dict(
    user_feed_class = UserPinFeed

    def add_pin(self, pin):
        activity = pin.create_activity()
        # add user activity adds it to the user feed, and starts the fanout
        self.add_user_activity(pin.user_id, activity)

    def get_user_follower_ids(self, user_id):
        ids = Follow.objects.filter(target=user_id).values_list('user_id', flat=True)
        return {FanoutPriority.HIGH:ids}

manager = PinManager()

Now that the manager class is set up, broadcasting a pin becomes as easy as:


Calling this method will insert the pin into your personal feed and into all the feeds of users which follow you. It does so by spawning many small tasks via Celery. In Django (or any other framework) you can now show the users feed.

# django example

def feed(request):
    Items pinned by the people you follow
    context = RequestContext(request)
    feed = manager.get_feeds(['normal']
    activities = list(feed[:25])
    context['activities'] = activities
    response = render_to_response('core/feed.html', context)
    return response

This example only briefly covered how Stream Framework works. The full explanation can be found on the documentation.


Stream Framework uses Celery and Redis/Cassandra to build a system with heavy writes and extremely light reads. It features:

  • Asynchronous tasks (All the heavy lifting happens in the background, your users don't wait for it)
  • Reusable components (You will need to make tradeoffs based on your use cases, Stream Framework doesn't get in your way)
  • Full Cassandra and Redis support
  • The Cassandra storage uses the new CQL3 and Python-Driver packages, which give you access to the latest Cassandra features.
  • Build for the extremely performant Cassandra 2.1. 2.2 and 3.3 also pass the test suite, but no production experience.
You can’t perform that action at this time.