Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Recipe like object extraction from HTML sources
Ruby
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
lib
spec
.autotest
.gitignore
README.rdoc
Rakefile
object-scraper.gemspec

README.rdoc

Object Scraper

Description

Object scraper is a thin wrapper for hpricot to enable receipt-like extraction of ruby objects from various web sites.

Install

Gem

gem install object-scraper --source http://gemcutter.org

Rails

config.gem 'object-scraper', :source => 'http://gemcutter.org'

Example

class Entry < Object
  attr_accessor :text, :date
end

uri     = "http://twitter.com/twitter"
pattern = ".status"

Scraper.define(:twitter, :class => :entry, :source => uri, :node => pattern) do |s|
  s.text { |node| node.at(".entry-content").inner_html }
  s.date { |node| DateTime.parse(node.at(".timestamp")[:data][/\'.*\'/].delete("'")) }
end

@objects = Scraper.parse(:twitter)

If you define multiple scrapers, you can collect all their objects with one simple method

@objects = Scraper.parse_all

Advanced Example

It is possible to use other existing HTML parsers instead of hpricot. Just overwrite the according proc object.

require 'nokogiri'
Scraper.scrape_source_with = Proc.new { |source| Nokogiri::HTML(source) }

Scraper.define(:twitter, :class => :entry, :source => uri, :node => pattern) do |s|
  # initialize your objects here accordingly
end

Rails

All scraper definitions sitting in RAILS_ROOT/scrapers will be taken into account automatically when you use object-scraper as a gem in your rails project.

Author

Something went wrong with that request. Please try again.