Skip to content
Scrapes a user's public tweets and saves them in JSON format.
Find file
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
bin
example
lib
.gitignore
README.md
package.json

README.md

Using Node.js and JQuery to Crawl Public Tweets

A talk by @benjamincoe

Birdeater

Birdeater is a command-line tool for backing up a user's public Tweets in JSON format.

Usage

First, make sure you have node package manager installed:

curl http://npmjs.org/install.sh | sh

Then run the following commands to install and run Birdeater:

npm install birdeater -g
birdeater --user=shitmydadsays

Be mindful when running it, as Twitter limits the number of requests that a single client can make per hour.

How Birdeater Works

Birdeater does not use Twitter's API. It was built as a demonstration of an approach I like to use for parsing structured information from unstructured HTML. Here's how it works:

An http connection is made to a user's public timeline using the request library:

User.prototype.loadStatuses = function(callback) {
    request({
        url: 'https://twitter.com/...',
        headers: {accept: 'application/json'}
    }, function(err, res, body) {
        var error = err || res.statusCode != 200;
        // handle error, or.
        callback( body );
    });
};

This returns an HTML representation of the tweets. JQuery is used to extract structured information from this:

jquery(body).find('.tweet').each(function() {
    var tweet = jquery(this);
    _this.tweets.push({
        text: jquery.trim( tweet.find('.js-tweet-text').text() ),
        timestamp: parseInt( tweet.find('._timestamp').attr('data-time') ),
        mentions: mentions.length > 0 ? tweet.attr('data-mentions').split(' ') : [],
        urls: urls
    });
});

I find that Node.js, coupled with JQuery, works great for building web crawlers:

  • Node.js offers a non-blocking evented paradigm. This is awesome for a web-crawler, which will tend to be I/O bound.
  • JQuery is a champ at parsing HTML. I find it much more intuitive than other libraries that I've used, such as Beautiful Soup.
  • First and foremost, it's easy to do (take a look at lib/user.js Birdeater clocks in at about 100 lines of code).

This approach has become my hammer when web scraping tasks come up.

-- Ben @benjamincoe

Something went wrong with that request. Please try again.