Guide: Harvesting social data

Amanda Hickman edited this page Aug 18, 2017 · 17 revisions


Mining data from social media platforms

At present, most journalists treat social sources like they would any other — individual anecdotes and single points of contact. But to do so with a handful of tweets and Instagram posts is to ignore the potential of hundreds of millions of others.

Many stories lay dormant in the vast amounts of data produced by everyday consumers. Here's a guide and tool box that may help you.

How to get the data

What data you can get with the scripts

This is a growing list of scripts we've put together to make social data mining easier. Right now we have scripts for Twitter and Facebook.


Before you begin

  1. If you don’t already have Python installed, start by getting Python up and running. Also have git installed.
  2. You should also make sure you have pip.

Twitter and Facebook-related preparations

  1. You need to get developer oauth credentials from the social media platforms you want to tap into. Oauth credentials are like an ID and password (often referred to as an app ID and secret respectively) that you create for an app or a script to access the data stream that a social media company provides. This data stream — also known as a company's Application Program Interface, or API — is often accessible using these credentials through a link (for example, this is what one of these queries could look like|YOURSECRET). Here's where you can get them: Twitter: Facebook:

Setting up the scripts

  1. Open up your Terminal and go to the folder where you want to clone this repository of code using the cd bash command.
git clone
cd social-data-scripts
  1. Then install all the dependencies, i.e. the Python libraries we are using for these scripts by running the following command:
pip install -r requirements.txt
  1. Make a file that is modeled after the file by going into the scripts directory and running these bash commands
cd scripts

Now you have a file! 🤗 Open it up in a text editor of your choice (like Atom or Sublime Text!) and fill the credentials you created earlier. Don't forget to save it!

Using Twitter's API


  • Up to 3200 tweets from an individual account (includes tweet id, time stamp, location, text, retweet count, favorite count (though the favorite count is inaccurate for retweets), whether something was a manual retweet, how it was tweeted (Tweetdek, Android, etc.)). This script was modified from @Yanofsky's original script.
  • Twitter user bios (name, display name, bio, followers count (at time of scraping), following count (at time of scraping), when the account was created, location given in the bio)
  • You can search Twitter via its search API going back 7 days and grab tweets (id, author name, timestamp when it was created, favorites (again, unreliable), retweets, text)


How to run each script

  1. Follow the instructions in the comments of each script to customize your API query and resulting .csv file
  2. Run your script with the bash command python to generate a csv of tweets or Facebook posts. Then, go make do some journalism-ing!
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.