Skip to content
Twitter as a Message Bus.
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
LICENSE
README
auth.py
expire.py
gist.py
pull.py
push.py
requirements.txt
stream.py

README

twitter-message-bus
-------------------

Twitter as a message bus.

INTRODUCTION
    This is a proof of concept to show how Twitter can be used as a global
    message bus.

    Twitter has a streaming service where a program can listen to tweets posted
    by a user; tweets based on a topic, location, etc. The same (more
    specifically a user-stream) is used here.

    A node posts a tweet on Twitter, through a registered account and all the
    nodes that have subscribed to that account's user-stream can listen to the
    messages. Since Twitter has a 140 character limit, we store the data on a
    GitHub Gist and tweet the ID of the gist.

    All the components in this program are loosely coupled, any customization
    can be plugged-in easily.


COMPONENTS
    [-] Authentication: Verifying the sender as well as the integrity of
        the transported messages is handled by Keybase.
    [-] Message Bus: Twitter handles transporting messages back-and-forth.
    [-] Data: Data is stored in an encrypted format (using Keybase) on GitHub.


REQUIREMENTS
    You will need a Linux machine which has redis, disque and python-2.7+
    installed.
    requirements.txt will install all the required Python libraries.


ARCHITECTURE
    [-] The disque job queue has two queues setup: 'in' and 'out'.
        The 'in' queue receives all the incoming tweets from Twitter; the 'out'
        queue contains tweet, gist ID's which will have a set expiry time.
        These tweets/gists will be tracked from here and deleted accordingly,
        once their TTL is expired.
    [-] A daemon will listen to the Twitter Streaming API and dumps tweets to
        the 'in' queue.
    [-] A daemon will listen to the 'out' queue and perform deletion of expired
        tweets and gists.
    [-] A daemon will listen to the 'in' queue (like a worker); reads and
        decrypts messages (using Keybase) by fetching data from gist(the tweet
        contains the ID of the gist).
    [-] A script will encrypt and sign the contents of the raw data.


MODULES
    This is a brief description of the Python modules in the source.
    [-] auth:   Logic for encryption, description, signing and verifying data.
    [-] gist:   CRUD operations on GitHub Gists.
    [-] stream: Listen to the Twitter Stream; dump tweet to the 'in' queue.
    [-] push:   Create a gist on GitHub (with the signed, encrypted) message,
                post the gist ID as a tweet.
    [-] pull:   Listen to the 'in' queue, fetch the contents of the gist (from
                ID); decrypt and verify the content.
    [-] expiry: Listen to the 'out' queue, keep a track of tweets/gists to be
                deleted, delete them when the expiration time has reached.


USAGE
    push.py [-h] [-s HOST:PORT [HOST:PORT ...]] [-d] -r KEYBASE-ID [-t N]
               (-i FILE | -m MESSAGE)

    Push data to the message bus.

    optional arguments:
    -h, --help            show this help message and exit
    -s HOST:PORT [HOST:PORT ...], --sockets HOST:PORT [HOST:PORT ...]
                        a list containing the host, port numbers to listen to;
                        defaults to localhost:7711 (for disque)
    -d, --debug           enable debugging
    -r KEYBASE-ID, --recipient KEYBASE-ID
                        keybase-id to send
    -t N, --ttl N         a TTL (in seconds) for the data on Twitter and
                          GitHub; if not specified, the data will remain
                          as a gist, tweet
    -i FILE, --in-file FILE
    -m MESSAGE, --message MESSAGE

--------------------------------------------------------------------------------

    pull.py [-h] [-s HOST:PORT [HOST:PORT ...]] [-d] [-r DELAY]

    Read messages from the message bus.

    optional arguments:
      -h, --help            show this help message and exit
      -s HOST:PORT [HOST:PORT ...], --sockets HOST:PORT [HOST:PORT ...]
                            a list containing the host, port numbers to listen
                            to; defaults to localhost:7711 (for disque)
      -d, --debug           enable debugging
      -r DELAY, --retry DELAY
                            queue check frequncy (in seconds); defaults to 8


--------------------------------------------------------------------------------

    stream.py [-h] [-s HOST:PORT [HOST:PORT ...]] -c CHANNEL [CHANNEL ...]
                     [-d]

    Listen to tweets; dump them to the queue.

    optional arguments:
      -h, --help            show this help message and exit
      -s HOST:PORT [HOST:PORT ...], --sockets HOST:PORT [HOST:PORT ...]
                            a list containing the host, port numbers to listen
                            to; defaults to localhost:7711 (for disque)
      -c CHANNEL [CHANNEL ...], --channels CHANNEL [CHANNEL ...]
                            Twitter accounts to follow
      -d, --debug           enable debugging


--------------------------------------------------------------------------------

    expire.py [-h] [-s HOST:PORT [HOST:PORT ...]] [-d] [-r DELAY]

    Delete gists, tweets if a TTL is set.

    optional arguments:
      -h, --help            show this help message and exit
      -s HOST:PORT [HOST:PORT ...], --sockets HOST:PORT [HOST:PORT ...]
                            a list containing the host, port numbers to listen
                            to; defaults to localhost:7711 (for disque)
      -d, --debug           enable debugging
      -r DELAY, --retry DELAY
                            queue check frequncy (in seconds); defaults to 8


--------------------------------------------------------------------------------


SETUP:
    redis:
        http://redis.io/topics/quickstart

    disque:
        https://github.com/antirez/disque#setup

    Keybase:
        https://keybase.io/docs/command_line
        Make sure you sign into Keybase and the client is up.

    Credentials:
        Setup a Twitter account, create an application and generate
        client-id, client-secret, access-token and access-token-secret.
        https://dev.twitter.com/oauth/overview/application-owner-access-tokens

        Setup a GitHub account, generate a personal access token;
        make sure 'gist' is in the list of selected scopes.
        https://git.io/vmNUX

        Store the API credentials in the source directory (after the clone)
        under vault/keys.json, in this format:

        {
            "github": "github-personal-access-token",
            "twitter": {
                "consumer-key": "twitter-app-consumer-key",
                "consumer-secret": "twitter-app-consumer-secret",
                "access-token": "twitter-app-access-token",
                "access-token-secret": "twitter-app-access-token-secret"
            }
        }

    twitter-message-bus (basic setup; check options for more functionality):
        $ git clone https://github.com/clickyotomy/twitter-message-bus \
          GIT_CLONE_DIR
        $ DISQUE_INSTALL_DIR/src/disque-server [OPTIONS]
        $ cd GIT_CLONE_DIR
        $ pip install -r requirements.txt
        $ ./stream.py -c 'twitter-handle'
        $ ./expiry.py
        $ ./pull.py
        $ ./push.py -m 'Do. Or do not. There is no try.' -r 'twitter-handle'

    # DISQUE_INSTALL_DIR: Path where disque was cloned and built.
    # GIT_CLONE_DIR: Path where you cloned this repository.


NOTES:
    [-] stream, expiry and pull run as daemons, you can pipe the output to a
        log-file to monitor them.
    [-] Keybase is still in alpha, so feel free to change the auth module.
    [-] As of now, there is support only for text/* mimetypes.
    [-] Since this is a proof of concept, the pull module does not do anything
        other than display the received message.
    [-] The source code is documented to the point.


May the Force be with you.
You can’t perform that action at this time.