Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Ruby and MongoDB integration for the back end of any new Twitter analytics/processing project

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 Gemfile
Octocat-spinner-32 Gemfile.lock
Octocat-spinner-32 Procfile
Octocat-spinner-32 README
Octocat-spinner-32 config.yaml
Octocat-spinner-32 tweeter-keeper.rb
README
Tweeter Keeper is built on top of other people's work - I didn't see any reason to reinvent the wheel.

TweetStream gem - reads tweets off of Twitter's streaming API and provides a Ruby block with the tweets.

Twitter gem - for use with the other Twitter API's - use to get followers for a user, user's tweet timeline, etc. We can use this to backfill in the history for users we follow on the streaming side. Some methods on this will require OAuth authentication

MongoDB - NoSQL database that uses BSON (similar to JSON) for inserting and retrieving data, making it easy to dump data directly from Twitter without processing. 

Mongo gem - Use the Mongo Ruby Driver directly, instead of an abstraction library

Yaml gem - Used for basic configuration tasks

# Install
=============

Environment - requires Ruby and MongoDB.

gem install bundle
bundle install


# Usage
==============

To run, open up config.yaml and change the USERNAME and PASSWORD to your Twitter account's username and password. 

bundle exec ruby tweeter-keeper.rb

Pick any tracking keywords you want, and add them to the @tracking_keywords array. 

This version has support for loading in one Twitter user's friends (the Twitter users they are following) and tracking their tweets as well - change the Twitter user's id from "jefflinwood" to your own or whomever. As long as the account isn't protected, you don't need the account's password or an OAuth authentication.

Open up a mongo shell and take a look at all the tweets you've collected! For sample data, I highly suggest "bieber" as a keyword.

> use tweeterkeeper

> db.tweets.count();
2156

# Future Plans
================
Get keywords directly from a MongoDB collection
Follow users from a MongoDB collection
Backfill user and keyword data using the REST API when a new user or keyword gets added
Handle errors/rate limits from the Twitter Streaming API
Auto-recovery if the stream gets dropped/errors
Something went wrong with that request. Please try again.