Skip to content
A Ruby library to parse the content out of web pages, such as BBC News pages. Used by the News Sniffer project.
Find file
Pull request Compare This branch is 68 commits behind johnl:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


Web Page Parser

Web Page Parser is a Ruby library to parse the content out of web pages, such as BBC News pages. It strips all non-textual stuff out, leaving the title, publication date and an array of paragraphs. It makes heavy use of regular expressions, rather than actually parsing the HTML. This may sound a bit whacky, but BBC News html in particular has semantic markup *within comments*, which cannot easily be referenced with standard HTML parsing. Regular expressions are much faster than full HTML parsing too.

Web Page Parser currently supports BBC News pages and Guardian news articles but new parsers are planned and can be added easily.

It is used by the News Sniffer project, which parses and archives news articles to keep track of how they change.

Example usage

require 'web-page-parser'
require 'open-uri'

url = ""
page_data = open(url).read

page = WebPageParser::ParserFactory.parser_for(:url => url, :page => page_data)

puts page.title # MPs hit back over expenses claims
puts # 2009-05-09T18:58:59+00:00
puts page.content.first # The wife of author Ken Follett and ...

More Info

Web Page Parser was written by John Leach and is released under the MIT License.

The code is available on github.

Something went wrong with that request. Please try again.