Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
simple python gevent web spider
Python
branch: master

change the SimpleHandler semantics to issue callbacks with only the j…

…ob, change the crawler to tack on the current crawler to the job during pre processing (so that job callbacks can access the crawler's job queue and add new ones independently of the handler on that job), change simple.startjobs to be able to create a simple handler from a callback so you can pass a BaseHandler or a callable as the handler kwarg
latest commit f3a869c602
@jmoiron authored

README.rst

aranha

Aranha (pronounced aranya) is a simple web spider written in python using gevent for asynchronicity. Aranha means "spider" in portuguese.

Aranha's goal is to be suitable for projects that need light url fetching or a simple spidering of a few classes of webpages. If spidering is a major part of your project, you probably want to either write your own spider or use scrapy as a base, as it's much more sophisticated.

Something went wrong with that request. Please try again.