psq / spider

Spider is a Web spidering library for Ruby. It handles the robots.txt, scraping, collecting, and looping so that you can just handle the data.

This URL has Read+Write access

The spider network graph

All branches in the network using psq/spider as the reference point. Read our blog post about how it works.

Show Help