Iron Crawler

A generic web crawler.

Features

From a starting URL, it will crawl all links on that URL and print a list of URLs visited.

• Follow href attributes contained in tags from the same domain
• Ignores href attributes contained in tags from other domains (even subdomains)
• Captures script src and link href tags for script and link tags respectively
• Outputs a list of visited URLs

Getting Started

It's easy to get started!

Install

gem install iron-crawler

Run

iron-crawler <url>

The above command will crawl any site for you.

TODO

• concurrency (will probably have to move away from mechanize)
• test coverage with Rspec
• set up CI pipeline with travis-ci to automatically publish to rubygems