Skip to content
Simple python script for storing tweets from the twitter stream directly to a MongoDB database based on a list of terms.
Python JavaScript
Find file
Latest commit 7bea4ff Apr 25, 2015 @gdelfresno Add follow option.
Get only tweets from certain users.
Failed to load latest commit information.
src Add follow option. Apr 25, 2015
.gitignore Remove .DS_Store Apr 6, 2013
LICENSE Tweepy, Oauth and License Oct 9, 2012
README.md Add follow option. Apr 25, 2015
requirements.txt Update and clean requirements Apr 23, 2015

README.md

twitterstream-to-mongodb

DESCRIPTION

Simple python script for storing tweets from the twitter stream directly to a MongoDB database based on a list of terms or users.

FEATURES/PROBLEMS

The script runs forever and refreshes the terms list periodically. Terms list can be modified while the scripts runs.

A catalog is created for each term in the MongoDB database.

Improvements apreciated.

CLONE AND USE

git clone git://github.com/gdelfresno/twitterstream-to-mongodb.git
cd twitterstream-to-mongodb/src
python twitterstreamtomongodb.py --oauth=oauth-example.json --server=localhost --port=23717 --database=TwitterStream --dbauth=dbauth.json --track=terms-example.txt --retweets=False

USAGE EXPLAINED

:arg oauth: json file that outlines oauth credentials for Twitter developers
:arg server: default is localhost for basic/local mongodb instances
:arg port: optional port of the mongodb instance
:arg database: the name you would like the database to have
:arg dbauth: auth file with database credentials
:arg track: basic text outlining search terms such as #trending or @user_name (carriage return per entry)
:arg follow: list of users to stream (without @)
:arg retweets: specify whether or not retweets are collected and stored in the database

--track and --follow can't be used at the same time.

DATABASE AUTH (json)

{
    "user" : "yor_user",
    "password" : "your_password"
}

OAUTH (json)

Oauth Authentication:

{
    "consumer_key" : "ThIsIsJuStAnExAmPlE",
    "consumer_secret" : "ThIsIsJuStAnExAmPlE",
    "access_token" : "ThIsIsJuStAnExAmPlE",
    "access_token_secret" : "ThIsIsJuStAnExAmPlE"
}

Basic Authentication:

{
    "username" : "twitter_username"
    "password" : "password"
}

TRACK (basic text)

SomeWord
@user_name
#hashtag

REQUIREMENTS

Install from requirements file

pip install -r requirements.txt

mongo-python-driver

https://github.com/mongodb/mongo-python-driver

pip install pymongo

If this doesn't work, install from source

git clone git://github.com/mongodb/mongo-python-driver.git pymongo
cd pymongo/
python setup.py install

tweepy

https://github.com/tweepy/tweepy

pip install tweepy

LICENSE:

Twitter Stream To MongoDB (c) by gdelfresno

Twitter Stream To MongoDB is licensed under 
the terms of the GNU General Public License 
as published by the Free Software Foundation.
Something went wrong with that request. Please try again.