Permalink
Fetching contributors…
Cannot retrieve contributors at this time
117 lines (77 sloc) 4.57 KB

MetaInspector

MetaInspector is a gem for web scraping purposes. You give it an URL, and it lets you easily get its title, links, and meta tags.

Installation

Install the gem from RubyGems:

gem install metainspector

This gem is tested on Ruby versions 1.8.7, 1.9.2 and 1.9.3.

Usage

Initialize a scraper instance for an URL, like this:

page = MetaInspector::Scraper.new('http://pagerankalert.com')

or, for short, a convenience alias is also available:

page = MetaInspector.new('http://pagerankalert.com')

If you don't include the scheme on the URL, http:// will be used by defaul:

page = MetaInspector.new('pagerankalert.com')

Then you can see the scraped data like this:

page.url                # URL of the page
page.scheme             # Scheme of the page (http, https)
page.title              # title of the page, as string
page.links              # array of strings, with every link found on the page
page.absolute_links     # array of all the links converted to absolute urls
page.meta_description   # meta description, as string
page.description        # returns the meta description, or the first long paragraph if no meta description is found
page.meta_keywords      # meta keywords, as string
page.image              # Most relevant image, if defined with og:image
page.images             # array of strings, with every img found on the page
page.absolute_images    # array of all the images converted to absolute urls
page.feed               # Get rss or atom links in meta data fields as array
page.meta_og_title      # opengraph title
page.meta_og_image      # opengraph image

MetaInspector uses dynamic methods for meta_tag discovery, so all these will work, and will be converted to a search of a meta tag by the corresponding name, and return its content attribute

page.meta_description       # <meta name="description" content="..." />
page.meta_keywords          # <meta name="keywords" content="..." />
page.meta_robots            # <meta name="robots" content="..." />
page.meta_generator         # <meta name="generator" content="..." />

It will also work for the meta tags of the form <meta http-equiv=“name” … />, like the following:

page.meta_content_language  # <meta http-equiv="content-language" content="..." />
page.meta_Content_Type      # <meta http-equiv="Content-Type" content="..." />

Please notice that MetaInspector is case sensitive, so page.meta_Content_Type is not the same as page.meta_content_type

You can also access most of the scraped data as a hash:

page.to_hash               # { "url"=>"http://pagerankalert.com", "title" => "PageRankAlert.com", ... }

The full scraped document if accessible from:

page.document # Nokogiri doc that you can use it to get any element from the page

Examples

You can find some sample scripts on the samples folder, including a basic scraping and a spider that will follow external links using a queue. What follows is an example of use from irb:

$ irb
>> require 'metainspector'
=> true

>> page = MetaInspector.new('http://pagerankalert.com')
=> #<MetaInspector:0x11330c0 @url="http://pagerankalert.com">

>> page.title
=> "PageRankAlert.com :: Track your PageRank changes"

>> page.meta_description
=> "Track your PageRank(TM) changes and receive alerts by email"

>> page.meta_keywords
=> "pagerank, seo, optimization, google"

>> page.links.size
=> 8

>> page.links[5]
=> "http://pagerankalert.posterous.com"

>> page.document.class
=> String

>> page.parsed_document.class
=> Nokogiri::HTML::Document

ZOMG Fork! Thank you!

You're welcome to fork this project and send pull requests. I want to thank specially:

To Do

Copyright © 2009-2011 Jaime Iniesta, released under the MIT license