Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
71 lines (52 sloc) 2.32 KB

Using Node.js and JQuery to Crawl Public Tweets

A talk by @benjamincoe


Birdeater is a command-line tool for backing up a user's public Tweets in JSON format.


First, make sure you have node package manager installed:

curl | sh

Then run the following commands to install and run Birdeater:

npm install birdeater -g
birdeater --user=shitmydadsays

Be mindful when running it, as Twitter limits the number of requests that a single client can make per hour.

How Birdeater Works

Birdeater does not use Twitter's API. It was built as a demonstration of an approach I like to use for parsing structured information from unstructured HTML. Here's how it works:

An http connection is made to a user's public timeline using the request library:

User.prototype.loadStatuses = function(callback) {
		url: '',
		headers: {accept: 'application/json'}
	}, function(err, res, body) {
		var error = err || res.statusCode != 200;
		// handle error, or.
		callback( body );

This returns an HTML representation of the tweets. JQuery is used to extract structured information from this:

jquery(body).find('.tweet').each(function() {
	var tweet = jquery(this);
		text: jquery.trim( tweet.find('.js-tweet-text').text() ),
		timestamp: parseInt( tweet.find('._timestamp').attr('data-time') ),
		mentions: mentions.length > 0 ? tweet.attr('data-mentions').split(' ') : [],
		urls: urls

I find that Node.js, coupled with JQuery, works great for building web crawlers:

  • Node.js offers a non-blocking evented paradigm. This is awesome for a web-crawler, which will tend to be I/O bound.
  • JQuery is a champ at parsing HTML. I find it much more intuitive than other libraries that I've used, such as Beautiful Soup.
  • First and foremost, it's easy to do (take a look at lib/user.js Birdeater clocks in at about 100 lines of code).

This approach has become my hammer when web scraping tasks come up.