Skip to content
Find file
Fetching contributors…
Cannot retrieve contributors at this time
28 lines (14 sloc) 770 Bytes


Experimental Distributed Web Crawling with Python + Gearman

Setup Instructions for Ubuntu:

$ sudo apt-get install git gearman libgearman-dev python-setuptools build-essential libxml2-dev libxslt-dev python-dev

$ sudo easy_install pyquery gearman tornado progressbar

If you are looking to do more than 1024 simultaneous connections on a single machine make sure you edit /etc/security/limits.conf and increase the soft/hard nofile limits.

Clone the Git Repo:

$ git clone git://
$ cd gearnado

Launch 30 TweetScout workers in one terminal:

$ for i in `seq 1 30`; do ./ & done

And run the TweetHandler in another:

$ time ./ --url_file=python_crawler_urls.txt
Something went wrong with that request. Please try again.