Skip to content
A Ruby library to parse the content out of web pages, such as BBC News pages. Used by the News Sniffer project.
Branch: master
Clone or download
Pull request Compare This branch is 48 commits behind johnl:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib
spec
.gitignore
Gemfile
Gemfile.lock
LICENSE
README.rdoc
Rakefile
web-page-parser.gemspec

README.rdoc

Web Page Parser

Web Page Parser is a Ruby library to parse the content out of web pages, such as BBC News pages. It strips all non-textual stuff out, leaving the title, publication date and an array of paragraphs. It makes heavy use of regular expressions, rather than actually parsing the HTML. This may sound a bit whacky, but BBC News html in particular has semantic markup *within comments*, which cannot easily be referenced with standard HTML parsing. Regular expressions are much faster than full HTML parsing too.

Web Page Parser currently supports BBC News pages and Guardian news articles but new parsers are planned and can be added easily.

It is used by the News Sniffer project, which parses and archives news articles to keep track of how they change.

Example usage

require 'web-page-parser'
require 'open-uri'

url = "http://news.bbc.co.uk/1/hi/uk/8041972.stm"
page_data = open(url).read

page = WebPageParser::ParserFactory.parser_for(:url => url, :page => page_data)

puts page.title # MPs hit back over expenses claims
puts page.date # 2009-05-09T18:58:59+00:00
puts page.content.first # The wife of author Ken Follett and ...

Ruby 1.8 support

Installing the Oniguruma gem on Ruby 1.8 will make Web Page Parser run faster, it's highly recommended but not required.

More Info

Web Page Parser was written by John Leach and is released under the MIT License.

The code is available on github.

You can’t perform that action at this time.