News Sniffer monitors news websites and detects when articles change. The versions are viewable and the changes are highlighted.
The main deployment is at http://www.newssniffer.co.uk/ which currently monitors a few key feeds from the BBC News website and the Guardian website, excluding a few sections like Sports (as they change too often with just scores).
The code is available under a free software license, so anybody can run their own deployment and monitor whatever news sources they wish (or just contribute to development).
News Sniffer uses the Ruby on Rails web framework. This document presumes you're familiar with that.
News Sniffer uses the
web-page-parser ruby library to parse the text
out of the news article html pages. If you want to add support for
other news sources, then start with adding it there:
Check out the code, and install the gems:
$ bundle install
configure a local mysql or sqlite development database and create the db and load the schema:
$ bundle exec rake db:create $ bundle exec rake db:schema:load
Then, using the rails console, add your first rss news feed. Let's add a BBC news feed:
$ bundle exec rails console irb(main):001:0> NewsArticleFeed.create :name => "BBC News", :url => "http://feeds.bbci.co.uk/news/rss.xml", :source => "bbc", :check_period => 300
:source option is an arbitrary key that is used to group several
feeds with the same source (and it's displayed on the article page).
:check_period options is the number of seconds to wait between
re-downloading the feed to check for new articles.
Then run the
newssniffer:articles:update rake task which will
download the feeds and create
NewsArticle records for any articles
$ bundle exec rake newssniffer:articles:update newssniffer:articles:update NewsArticleFeed 1 NewsArticleFeed 1, 79 new articles discovered
Then run the
newssniffer:versions:update rake task, which will
download the page content for the news articles:
$ bundle exec rake newssniffer:versions:update newssniffer:versions:update NewsArticle 1 new version found 1 NewsArticle 2 new version found 2 NewsArticle 3 new version found 3
You can run these rake tasks as often as you like - only the feeds and articles that are overdue a check are processed.
News Sniffer supports filtering rss feeds, so you can avoid tracking certain articles. Currently it supports filtering by regular expressions applied to the urls (used by www.newssniffer.co.uk to filter out certain articles from the BBC, such as sport articles which change a lot and aren't usually interesting politically).
You specify filters by creating
$ NewsArticleFeedFilter.create :name => "Ignore BBC sport articles", :url_filter => 'sport/0'
Author:: John Leach (mailto:email@example.com) Copyright:: Copyright (c) 2006-2012 John Leach License:: GNU Affero General Public License v3 Web page:: http://www.newssniffer.co.uk Github:: http://github.com/johnl/news-sniffer