A component of the EDGI Web Monitoring Project.
Overview of this component's tasks
This component is intended to hold various backend tools serving different tasks:
- Query external sources of captured web pages (e.g. Internet Archive, Page Freezer, Sentry), and formulate a request for importing their version and page metadata into web-monitoring-db.
- Provide a web service that computes the "diff" between two versions of a page in response to a query from web-monitoring-db.
- Query web-monitoring-db for new Changes, analyze them in an automated pipeline to assign priority and/or filter out uninteresting ones, and submit this information back to web-monitoring-db.
Working and Under Active Development:
- A Python API to PageFreezer's diffing service in
- A Python API to the Internet Archive Wayback Machine's archived webpage
- A Python API to the web-monitoring-db Rails app in
- Python functions and a command-line tool for importing snapshots from PF and IA into web-monitoring-db.
Legacy projects that may be revisited:
- Example HTML providing useful test cases.
Get Python 3.6. This packages makes use of modern Python features and requires Python 3.6+. If you don't have Python 3.6, we recommend using conda to install it. (You don't need admin privileges to install or use it, and it won't interfere with any other installations of Python already on your system.)
Install the package.
pip install -r requirements.txt python setup.py develop
Copy the script
.envand supply any local configuration info you need. (Only some of the package's functionality requires this.) Apply the configuration:
See module comments and docstrings for more usage information. Also see the command line tool
wm, which is installed with the package. For help, use
To run the tests or build the documentation, first install the development requirements.
pip install -r dev-requirements.txt
To build the docs:
cd docs make html
To run the tests:
Any additional arguments are passed through to
The Dockerfile runs
wm-diffing-server on port 80 in the container. To build
docker build -t processing . docker run -p 4000:80 processing
Point your browser or
Code of Conduct
This repository falls under EDGI's Code of Conduct.
This project wouldn’t exist without a lot of amazing people’s help. Thanks to the following for all their contributions! See our contributing guidelines to find out how you can help.
|Chaitanya Prakash Bapat|
|Janak Raj Chadha|
(For a key to the contribution emoji or more info on this format, check out “All Contributors.”)
License & Copyright
Copyright (C) 2017-2018 Environmental Data and Governance Initiative (EDGI)
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.0.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
LICENSE file for details.