RW Spider is an multithreading spider client written in Ruby. The library was designed to make easy the development of programs that spider the web.
RW Spider design comes from the direct experiences of the development of another PHP library that is currently used as the engine for an freeware SEO tool.
Customizable options for the spider job
Ruby >= 1.8.7
Hpricot >= 0.8.2
Robotstxt >= 0.5.2
This library is intended to be installed via the Gemcutter system.
$ gem install rwspider
You might need administrator privileges on your system to install it.
Simone Rinzivillo <firstname.lastname@example.org>
See the CHANGELOG.rdoc file for details.
Copyright © 2009 Simone Rinzivillo, RW Spider is released under the MIT license.