Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Twitter User Timeline Harvest
Branch: master
Failed to load latest commit information.
.gitignore removed folders
MANIFEST sdist updated readme removed packages from setup Changed copyright - 2015


Twitter-Harvest is a Twitter User Timeline tool that allows the user to download statuses (tweets) from user timelines as JSON objects and store them into a MongoDB database.

Developed on Python 2.7.x, you must also have Twitter Auth Credentials (detailed below).




python install
  • uses setuptools!



python --consumer-key consumer-key --consumer-secret consumer-secret --access-token access-token --access-secret access-secret --db mongodb-uri
  • The consumer-key, consumer-secret, access-token and access-secret arguments are required.

Other Useful Options

-r include native retweets in harvest (default = False)

-v print in stdout all the tweets being harvested (default = False)

--db connection URI, allows for insertion into a MongoDB

--numtweets total number of tweets to be harvested (max = 3200)

--user user's timeline you would like to harvest. must be public or already following. (default = mongolab)

Help Contents

usage: [-h] [-r] [-v] [--numtweets NUMTWEETS] [--user USER]
                          --db DB --consumer-key CONSUMER_KEY
                          --consumer-secret CONSUMER_SECRET 
                          --access-token ACCESS_TOKEN 
                          --access-secret ACCESS_SECRET

Connects to Twitter User Timeline endpoint, retrieves tweets and inserts into
a MongoDB database. Developed on Python 2.7

optional arguments:
  -h, --help                          show this help message and exit
  -r, --retweet                       include native retweets in the harvest
  -v, --verbose                       print harvested tweets in shell
  --numtweets NUMTWEETS               set total number of tweets to be harvested, max = 3200
  --user USER                         choose user timeline for harvest
  --db DB                             MongoDB URI, example: mongodb://
  --consumer-key CONSUMER_KEY         Consumer Key from your Twitter App OAuth settings
  --consumer-secret CONSUMER_SECRET   Consumer Secret from your Twitter App OAuth settings
  --access-token ACCESS_TOKEN         Access Token from your Twitter App OAuth settings
  --access-secret ACCESS_SECRET       Access Token Secret from your Twitter App Dev Credentials

Twitter App Setup

For those unfamiliar with the Twitter Dev/App page, here are instructions for getting this script up and running.

  1. Visit the Twitter dev page.
  2. Go to My Applications (should be a drop down from your username).
  3. Create new app, fill in required fields.
  4. Your access token, access token secret, consumer key, and consumer secret should all be displayed. Assign those values accordingly in the script :)


Feel free to contact me via twitter @chrisckchang, email, or drop us a line at if you have any questions or comments!

Something went wrong with that request. Please try again.