Rcrawl web crawler for Ruby

Rcrawl is a web crawler written entirely in ruby. It's limited right now by the fact that it will stay on the original domain provided.

Rcrawl is available as a gem here: https://rubygems.org/gems/rcrawl/versions/0.5.1

Rcrawl uses a modular approach to processing the HTML it receives. The link exraction portion of Rcrawl depends on the scrAPI toolkit by Assaf Arkin (http://labnotes.org). gem install scrapi

The structure of the crawling process was inspired by the specs of the Mercator crawler (http://www.cindoc.csic.es/cybermetrics/pdf/68.pdf). A (somewhat) brief overview of the design philosophy follows.

The Rcrawl process

1. Remove an absolute URL from the URL Server.
2. Download corresponding document from the internet, grabbing and processing robots.txt first, if available.
3. Feed the document into a rewind input stream(ris) to be read/re-read as needed. Based on MIME type, invoke the process method of the
4. processing module associated with that MIME type. For example, a link extractor or tag counter module for text/html MIME types, or a gif stats module for image/gif. By default, all text/html MIME types will pass through the link extractor. Each link will be converted to an absolute URL and tested against a (ideally user-supplied) URL filter to determine if it should be downloaded.
5. If the URL passes the filter (currently hard coded as Same Domain?), then call the URL-seen? test.
6. Has the URL been seen before? Namely, is it in the URL Server or has it been downloaded already? If the URL is new, it is added to the URL Server.
7. Back to step 1, repeat until the URL Server is empty.

Examples

Instantiate a new Rcrawl object

`crawler = Rcrawl::Crawler.new(url)`

Begin the crawl process

`crawler.crawl`

After the crawler is done crawling

Returns an array of visited links

`crawler.visited_links`

Returns a hash where the key is a url and the value is the raw html from that url

`crawler.raw_html`

Returns a hash where the key is a URL and the value is

the error message from stderr

`crawler.errors`

Returns an array of external links

`crawler.external_links`

Set user agent

`crawler.user_agent = "Your fancy crawler name here"`

License

Copyright © 2006 Digital Duckies, LLC, under MIT License

Developed for Digital Duckies by Shawn Hansen and Jason Burgett