Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
RW Spider is an multithreading spider client written in Ruby designed to make easy the development of programs that spider the web.
Ruby
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
lib
test
Changelog.rdoc
LICENSE.rdoc
Manifest
README.rdoc
Rakefile

README.rdoc

RW Spider

RW Spider is an multithreading spider client written in Ruby. The library was designed to make easy the development of programs that spider the web.

RW Spider design comes from the direct experiences of the development of another PHP library that is currently used as the engine for an freeware SEO tool.

Features

  • Multithreading spider

  • Customizable options for the spider job

  • Robots.txt support

  • Indexing of web pages and others files (images, CSS, JavaScript, PDF and more)

  • Following redirects

Requirements

  • Ruby >= 1.8.7

  • Hpricot >= 0.8.2

  • Robotstxt >= 0.5.2

Installation

This library is intended to be installed via the Gemcutter system.

$ gem install rwspider

You might need administrator privileges on your system to install it.

Author

Author

Simone Rinzivillo <srinzivillo@gmail.com>

Resources

Changelog

See the CHANGELOG.rdoc file for details.

License

Copyright © 2009 Simone Rinzivillo, RW Spider is released under the MIT license.

Something went wrong with that request. Please try again.