Skip to content
A short and simple python crawler, that uses Webkit and executes Javascript
Latest commit 5a9abb5 Jan 24, 2013 @invernizzi committed with ''better'' waiting
Failed to load latest commit information.
.gitignore Initial commit Jun 19, 2012 Update master Jun 19, 2012 ''better'' waiting Jan 24, 2013


A short and simple web crawler written in Python, that uses Webkit and executes Javascript.

How to use

crawler = Crawler(gui=True,                                                 # To see the crawler in action
                  is_link_interesting=lambda url, text: 'download' in url)  # Follow every link containing
                                                                            #  "download" in the url
Something went wrong with that request. Please try again.