Stream Spigot a collection of tools to make consumption of real time-ish datasources more manageable. Four tools have been built, but only Masto Feeder is still active:
- Tweet Digest lets tweets be consumed in daily digests (ideally via a feed reader). This is useful for high-volume accounts that would be annoying to follow directly. Digests can be defined either with explicit lists of usernames or by using the built-in Twitter list functionality.
- Feed Playback allows RSS/Atom feed content to be played back (Google Reader is used for historical data). This makes it easy to read a newly-discovered blog from the beginning in a manageable fashion.
- Bird Feeder is similar to Tweet Digest, but instead of daily digests, it lets you read your Twitter timeline in a feed reader. This is useful for people who follow a lot of people and don't want to miss anything.
- Masto Feeder is similar to Bird Feeder, but for Mastodon timelines. It's the only tool that's still active.
A running instance is at www.streamspigot.com.
The various tools (tweet digest, feed playback, etc.) are pretty distinct. Parallel directories are maintained in app
and app/templates
(e.g. app/tweetdigest
and app/templates/tweetdigest
). Within each tools the convention is to have a handlers.py
with all the HTTP request handlers and a data.py
with all the business logic (non-tool specific API bindings go into datasources
, see below for more).
Base/utility code goes in app/base
and app/templates/base
.
UI JavaScript is minimal, and is composed of a few modules in app/static
.
The tweet digest component of the app was formerly known as Twitter Digest and used to run at twitter-digest.appspot.com
. A small "stub" App Engine app that redirects requests from there to www.streamspigot.com/tweet-digest
is in twitter-digest-stub
.
Third-party API endpoints are in the datasources
directory:
twitter.py
is the Python wrapper for the Twitter API provided by http://code.google.com/p/python-twitter/ (version 0.8.1 with some small local modifications). It uses authentication even for requests for publicly-visible data so that we don't get rate-limited as often.googlereader.py
is a simple Python wrapper for the Google Reader "API" methdos that we need. Some methods require authentication; for now we don't act on behalf of users, instead the app has a few designated Google accounts that it stores data it.
For both Twitter and Google Reader, authentication is done via OAuth, the access tokens are stored in oauth_keys.py
(the file is not checked in, but can
be created based on the oauth_keys.py.template
file with the help of get_oauth_access_token.py
and tested with test_oauth_access_token.py
).
The top-level Makefile
file has a couple of helper commands, dev
to run with the App Engine development server and deploy
to deploy the app to production.