Skip to content
A short and simple python crawler, that uses Webkit and executes Javascript
Python
Latest commit 5a9abb5 Jan 24, 2013 @invernizzi committed with ''better'' waiting
Failed to load latest commit information.
.gitignore Initial commit Jun 19, 2012
README.md Update master Jun 19, 2012
crawler.py ''better'' waiting Jan 24, 2013

README.md

js-crawler

A short and simple web crawler written in Python, that uses Webkit and executes Javascript.

How to use

crawler = Crawler(gui=True,                                                 # To see the crawler in action
                  is_link_interesting=lambda url, text: 'download' in url)  # Follow every link containing
                                                                            #  "download" in the url
crawler.crawl('http://firefox.com')
crawler.close()
Something went wrong with that request. Please try again.