A very simple web crawler written in Ruby. Useful to build Sitemaps and please your SEO obssessed friends.
Ruby
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.gitignore
MIT_LICENCE
README.rdoc
Rakefile
discovery_mission.gemspec
discovery_mission.rb
spec.rb

README.rdoc

Discovery Mission

A SPACE ODYSSEY

This is just a simple, basic (dummy?) and lightweight crawler. It is used to reference all URLs of your website in order to record them on a database, and use them for a sitemap manager.

Just install it:

# sudo gem install discovery-mission

And then you're ready to take off:

require 'discovery_mission'

# Use a block to do something with each path # response is a 'net/http' response object, therefore responds to :body, :code …

DiscoveryMission.for(“www.my-domain.com”) do |url, response| puts “Dave Bowman landed on #{url}” case response when Net::HTTPSuccess puts “Fuck you HAL 9000” when Net::HTTPRedirection puts “Shit! That's Jupiter!” else puts “Bad trip indeed” end end

# Or just without a block because an Array with all paths is returned

my_sitemap_entries = DiscoveryMission.for(“www.my-domain.com”)

-

Copyright © 2009 - 2010 Mickael Riga. See MIT_LICENCE file for details.