Web Page Parser is a Ruby library to parse the content out of web pages, such as BBC News pages. It strips all non-textual stuff out, leaving the title, publication date and an array of paragraphs. It makes heavy use of regular expressions, rather than actually parsing the HTML. This may sound a bit whacky, but BBC News html in particular has semantic markup *within comments*, which cannot easily be referenced with standard HTML parsing. Regular expressions are much faster than full HTML parsing too.
Web Page Parser currently supports BBC News pages and Guardian news articles but new parsers are planned and can be added easily.
It is used by the News Sniffer project, which parses and archives news articles to keep track of how they change.
require 'web-page-parser' require 'open-uri' url = "http://news.bbc.co.uk/1/hi/uk/8041972.stm" page_data = open(url).read page = WebPageParser::ParserFactory.parser_for(:url => url, :page => page_data) puts page.title # MPs hit back over expenses claims puts page.date # 2009-05-09T18:58:59+00:00 puts page.content.first # The wife of author Ken Follett and ...
Web Page Parser was written by John Leach and is released under the MIT License.
The code is available on github.