Automatic scraper that tracks changes in news articles over time.
Switch branches/tags
Nothing to show
Clone or download
ecprice Merge pull request #49 from carlgieringer/master
Fix NYTParser for new article format
Latest commit 279e867 Jun 16, 2018


A website and framework that tracks changes in online news articles over time.

Original installation at A product of the Knight Mozilla MIT news hackathon in June 2012. Authors: Eric Price (, Greg Price (, and Jennifer 8. Lee (

This is free software under the MIT/Expat license; see LICENSE. The project's source code lives at .


You need to have installed on your local machine

  • Git
  • Python 2.6 or later
  • Django and other Python libraries

On a Debian- or Ubuntu-based system, it may suffice (untested) to run $ sudo apt-get install git-core python-django python-django-south python-simplejson

On Mac OS, the easiest way may be to install pip: and then $ pip install Django

Initial setup

$ python website/ syncdb && python website/ migrate $ mkdir articles

Running NewsDiffs Locally

Do the initial setup above. Then to start the webserver for testing: $ python website/ runserver

and visit http://localhost:8000/

Running the scraper

Do the initial setup above. You will also need additional Python libraries; on a Debian- or Ubuntu-based system, it may suffice (untested) to run $ sudo apt-get install python-bs4 python-beautifulsoup

on a Mac, you will want something like

$ pip install beautifulsoup4 $ pip install beautifulsoup $ pip install html5lib

Note that we need two versions of BeautifulSoup, both 3.2 and 4.0; some websites are parsed correctly in only one version.

Then run $ python website/ scraper

This will populate the articles repository with a list of current news articles. This is a snapshot at a single time, so the website will not yet have any changes. To get changes, wait some time (say, 3 hours) and run 'python website/ scraper' again. If any of the articles have changed in the intervening time, the website should display the associated changes.

The scraper will log progress to /tmp/newsdiffs_logging (which is overwritten each run) and errors to /tmp/newsdiffs/logging_errs (which is cumulative).

To run the scraper every hour, run something like:

$ while true; do python website/ scraper; sleep 60m; done

or make a cron job.

Adding new sites to the scraper

The procedure for adding new sites to the scraper is outlined in parsers/ . You need to

(1) Create a new parser module in parsers/ . This should be a subclass of BaseParser (in parsers/ Model it off the other parsers in that directory. You can test the parser with by running, e.g.,

$ python parsers/ bbc.BBCParser

  which will output a list of URLs to track, and

$ python parsers/ bbc.BBCParser

  which will output the text that NewsDiffs would store.

(2) Add the parser to 'parsers' in parsers/

This should cause the scraper to start tracking the site.

To make the source display properly on the website, you will need minor edits to two other files: website/frontend/ and website/frontend/ (to define the display name and create a tab for the source, respectively). Search for 'bbc' to find the locations to edit.