Simple, fast user news feeds for Django
Author: Jamie Matthews. Follow me on Twitter.
- Django 1.3, 1.4, 1.5
An app for creating news feeds (also known as activity streams) for users, notifying them of activity happening around your site. Optimised for speed, pluggability and simplicity.
News feed items are stored as a string and a timestamp. You can't store any
additional metadata about the stream items, such as generic foreign keys to and
Actor or a
Target. You just store the item content as plain text (or HTML).
If you need links to other objects, just insert an
You can install django-user-streams from PyPI:
pip install django-user-streams
user_streams to your
INSTALLED_APPS setting. You also need a backend,
which defines how your streams are stored. These are described below.
INSTALLED_APPS = [ ... 'user_streams', 'user_streams.backends.user_streams_single_table_backend', ... ] USER_STREAMS_BACKEND = 'user_streams.backends.user_streams_single_table_backend.SingleTableDatabaseBackend'
Finally, if you're using a backend that stores stream items using Django's model
manage.py syncdb to create the necessary database tables.
Adding items to streams
To create a stream item:
import user_streams user = User.objects.get(username='jamie') user_streams.add_stream_item(user, 'This is the contents of the stream item')
The first argument to
add_stream_item can be a single
User instance, or a queryset
representing multiple users. In the latter case, the message you supply is added
to the stream of each user in the queryset.
import user_streams user_streams.add_stream_item(User.objects.all(), 'Broadcast message to all users')
You can also specify the creation time for the stream item by passing a
datetime.datetime instance as the value of the
python import user_streams from datetime import datetime user = User.objects.get(username='jamie') user_streams.add_stream_item(user, 'You have a new message!', created_at=datetime.now())
A note on time zones
When a stream item is created, Django's timezone support settings will be respected.
If timezone support is enabled by setting
True, then timezone-aware datestamps will be used, and stream items will be stored in the database using a UTC offset. You will need to convert the timestamps to your users' local time at the last possible moment (when the
datetime object is formatted for presentation to the user).
If timezone support is disabled by setting
False, then timezone-naive datestamps will be used, and stream items should be dealt with as using localtime.
Time zones and Django 1.3 compatibility
Django's timezone support was added in 1.4, so things work a little differently if you're using
django-user-streams with Django 1.3.
By default, if you don't pass a
created_at argument to
datetime.datetime.now() will be used to timestamp your stream items.
This is probably the least surprising behaviour, and if your app only ever deals
with users in one timezone (and those users are in the same timezone as your
web server), it's probably fine.
If your users are all over the world, however, this is a bad idea. The reasons
for this are discussed in
this blog post by Armin Ronacher.
The best way to store timestamps in the database is to use the UTC timezone.
You can then convert them to your users' local time at the last possible moment
datetime object is formatted for presentation to the user).
To support this, you can either provide the
created_at argument every time
you call the
user_streams.add_stream_item(user, 'You have a new message!', created_at=datetime.utcnow())
Alternatively, you can set the
USER_STREAMS_USE_UTC setting (in your
False by default). If you do this,
datetime.utcnow() will be used instead of
datetime.now() to generate
the timestamps for each stream item.
If you do either of these things, the
created_at property of each of your
stream items will be set to UTC time. It's your responsibility to convert
this to each user's local time for formatting. Take a look at
times for an easy way to deal with that.
Support for Django 1.3 and the
USER_STREAMS_USE_UTC setting is intended to be deperecated at some point in the future.
Getting the stream for a user
To retrieve the stream items for a user:
import user_streams user = User.objects.get(username='jamie') items = user_streams.get_stream_items(user)
This will return an iterable of objects, each of which is guaranteed to have two
created_at, which will be a
representing the creation timestamp of the message, and
content, which will
contain the contents of the message as a string. The objects will be ordered by
created_at field, with the most recent first. The iterable that is
returned will be lazy, meaning that you can slice it (and pass it to a Django
Paginator object) without loading all of the items from the database.
Stream storage is abstracted into
Backend classes. Three backends are
django-user-streams. Each backend is kept in a separate
reusable app, which must be added to
INSTALLED_APPS separately to the main
user_streams app. This is to ensure that only the database tables required
for each backend are created (assuming you are using a backend that stores
data through Django's model layer).
Which backend you choose depends on the scale of your application, as well as your expected usage patterns. The pros and cons of each are described below.
The simplest backend. Your stream items are stored in a single database table,
consisting of a foreign key to a
User object, a
DateTimeField timestamp, and
TextField to store your message. Fetching a stream for a user should be
extremely fast, as no database joins are involved. The tradeoff is storage
space: If you send a message to multiple users, the message is stored multiple
times, once for each user. If you regularly broadcast messages to thousands of
users, you may find that the table gets very large.
This backend stores your messages in a table with a
relationship to your
User objects. Each message is only stored once, with a
row in the intermediate table for each recipient. This means you need much less
space for broadcast messages, but your queries may be slightly slower.
Stores your messages in Redis sorted sets, one set for each user, with a Unix
created_at attribute) as the score for each item. This approach
is described in more detail
iterable returned by
ZREVRANGE to retrieve each slice of the
ZCARD to get the complete size of the set of items. This backend
should be screamingly fast.
Note: the Redis backend requires the
redis-py library. Install with
Redis backend settings
The following settings control the behaviour of the Redis backend:
Each key generated by the backend will be prefixed with the value of this setting. The default prefix is "user_streams".
A dictionary of keyword arguments which will be passed to the constructor of the Redis client instance.
Writing your own backend
You can create your own backend to store messages in whatever data store suits your application. Backends are simple classes which must implement two methods:
add_stream_item(self, users, content, created_at)
users will be an iterable of
User instances (you don't need to worry
about accepting a single instance - your backend method will always be called
with an iterable, which may be a list containing only one
content will be a string containing the stream message to store.
created_at will be a Python
datetime.datetime object representing the
time at which the stream item was created.
This method should return an iterable of messages for the given
by timestamp with the newest first. Each item must be an object with two
created_at (which must be a Python
content (which must be a string containing the message contents).
While this method could simply return a
list of messages, it's much more
efficient to assume that the list will be paginated in some way, and support
slicing and counting the objects on-demand, in whatever method your data store
supports. To do this, you should return an iterable object, overriding
__len__. See the implementation of
RedisBackend for an
To contribute: fork the repository, make your changes, add some tests, commit, push to a feature branch, and open a pull request.
How to run the tests
Clone the repo, install the requirements into your virtualenv, then type
python manage.py test user_streams. You can also use
python manage.py test user_streams user_streams_single_table_backend user_streams_many_to_many_backend user_streams_redis_backend to
run the tests for all the backends. Any of the above should also work if
django-user-streams into an existing Django project (of
course, only run the tests for the backend you're using).
- Add compatibility with Django 1.4's support for timezones
- Backends renamed to make app_labels less generic (for example,
- Add tests for pagination of results
- Fix result loading in RedisBackend
- Fix slicing behaviour in Redis backend
- Fix packaging
- Initial release.
Copyright (c) DabApps All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The views and conclusions contained in the software and documentation are those of the authors and should not be interpreted as representing official policies, either expressed or implied, of DabApps.