OpenStreetMap Analytic Difference Engine
Python HTML CSS JavaScript Dockerfile Shell Smarty
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
apiserver Add API server Jun 24, 2017
doc Update architecture drawings Jan 20, 2018
docker Add marking of review requests and bot users May 28, 2018
grafana Grafafa dashbord refactored using grafanalib Jul 1, 2018
html Add marking of review requests and bot users May 28, 2018
kubernetes Update nginx version Aug 16, 2018
osm Add marking of review requests and bot users May 28, 2018
schemas Add messagebus, some refactoring included Jan 14, 2018
templates Add marking of review requests and bot users May 28, 2018
test Add eventlet timeout May 31, 2018
.gitignore Add messagebus, some refactoring included Jan 14, 2018
.travis.yml Remove API server tests from TravisCI config Jun 24, 2017
Backend.py Reduce default horizon to 24hrs Feb 23, 2018
BackendDetailed.py Reduce default horizon to 24hrs Feb 23, 2018
BackendGeoJson.py Reduce default horizon to 24hrs Feb 23, 2018
BackendHtml.py Reduce default horizon to 24hrs Feb 23, 2018
BackendHtmlSummary.py Improve logging Jun 3, 2018
ColourScheme.py Initial version for github Sep 12, 2015
HumanTime.py Initial version for github Sep 12, 2015
LICENSE Initial commit Sep 12, 2015
README.md Update Kubernetes link Jun 30, 2018
apiserver.py Update cset refreshing, add more Prometheus metrics Jan 28, 2018
config.json Increase timeout for cset processing Jun 3, 2018
config.py Changed python interpreter path to use env Sep 18, 2016
db.py Handle mongodb size errors Jun 4, 2018
elastic-gw.py Update cset refreshing, add more Prometheus metrics Jan 28, 2018
icon.svg Improved documentation, added icon files, added csetinfo.py test appl… Sep 13, 2015
icon_bw.svg Major refactoring. State tracked in mongodb, monolitic tracker split … Aug 8, 2016
logging.conf Add backend metric, extend histogram time-span, simplify backend work… Feb 17, 2018
messagebus.py Enable auto-delete on AMQP queues Jun 11, 2018
osmtracker.py Enable auto-delete on AMQP queues Jun 11, 2018
requirements.txt Update python dependencies May 21, 2018
tempfilewriter.py Added label filtering feature, add tests Sep 14, 2016
test-requirements.txt Update Python requirements Jun 24, 2017

README.md

OpenStreetMap Analytic Difference Engine

Build Status

OpenStreetMap Analytic Difference Engine is an analytic live (web) service for OpenStreetMap (OSM) edits. Edits are analysed and presented with a concise summary. The (text-based) summary is designed to provide a good insight into what each changeset comtains (what was added, changed or deleted) and an optional visual-diff provides insight into the geographical changes of each changeset.

This live service is different from other OpenStreetMap live services in that it focus on being analytic and less on being visual.

A running demo can be seen on https://osm.expandable.dk. Read below for the details about the service. For deployment, see the Kubernetes deployment.

The purpose of this tool is:

  1. Prof-of-concept for an improved changeset-info service (improved compared to looking up changeset details on http://openstreetmap.org)
  2. Provide insight into changes in your area of interest
  3. Improve team spirit of a regional OSM team/task-force.
  4. Quality ensurance through peer review
  5. Learning by seeing how other make edits to your region of interest.

The service is available as a web-service and provide three different information elements:

  1. An overall summary with a overview map containing bounding boxes of recent edits
  2. A list of changesets with analytic details
  3. A visual diff for the changes of each changeset

Summary

The main page contain this overview in the top with OSM-user specific colour-coded bounding boxes for each changeset and text-based summary for the tracked time peried.

Image

List of Changesets with changes

Green show added tags and how many additions the changeset contained. Yellow show changed tags and red show deleted tags.

Image

Visual Diff

Geographical changes in the changeset, green are added objects, blue changed and red deleted. Each element can be clicked for more details.

Image Image

How Does It Work?

TL;DR:

Run it using Docker:

docker run -p 8080:80 -e OSMTRACKER_REGION="/osm-regions/denmark.poly" -e OSMTRACKER_MAP_SCALE="6" michaelvl/osmtracker-all-in-one

and point your browser to 127.0.0.1:8080. This docker image contain all necessary services to demonstrate the analytic tracker, including two worker instances. It is however not a good example of a 'production' deployment. For production-class deployments take a look at the Kubernetes deployment in the kubernetes folder.

Using an alternative config

The multi-dimensional labeling system described below allows for quite advanced configurations, however, the most typical configuration is to change the focus region and the scale of the default map. This can be done by setting the environment variables OSMTRACKER_REGION and OSMTRACKER_MAP_SCALE. See the regions.txt for available regions in the pre-build images.

Using a full custom config file can be done as follows:

curl -O https://raw.githubusercontent.com/MichaelVL/osm-analytic-tracker/master/config.json
mkdir osmtracker-config
mv config.json osmtracker-config/
sed -i 's/"path": "html"/"path": "\/html\/dynamic"/' osmtracker-config/config.json
docker run -p 8080:80 -v <path>/osmtracker-config:/osmtracker-config osmtracker-all-in-one

Labels and filters

Changesets are filtered using a labeling system and labels can also be used in backends for generating different views into the changesets. The following excerpt from the config file illustrates some possibilities with labels. Initially the 'pre_labels' section is executed. Currently two types are tests are implemented; An area-based test and a regular expression-based test (which can test on all fields of a changeset, including tags on changes). If an area or regex test is found be succeed, the associated label is applied to the changeset.

The config below specify to mark changesets inside 'region.poly' with labels 'inside-area' and 'center-inside-area' (with a small difference in how 'inside' is defined). Also, if the changeset comment has a '#' followed by some text, it will be labeled with 'mapping-event'.

The definition of 'prefilter_labels' below defines that all changesets which do not have the three labels are to be dropped. Generally the 'prefilter_labels' definition use an AND between the inner list and an OR on the outer list, i.e. '[['foo', 'bar'], ['baz']]' would mean that a changeset should have labels 'foo' and 'bar' OR 'baz'.

	"pre_labels": [
	    {"area_file": "region.poly", "area_check_type": "cset-bbox", "label": "inside-area"},
	    {"area_file": "region.poly", "area_check_type": "cset-center", "label": "center-inside-area"},
	    {"regex": [{".meta.tag.comment": ".*#\\w+"}], "label": "mapping-event"}
	],
	"prefilter_labels": [["inside-area", "center-inside-area", "mapping-event"]],

Note that 'prefilter_labels' regex can only operate on changeset metadata. Labeling on changeset content is done using 'post_labels' as seen from the following config except. This labels changesets which has changes with tag 'osak:identifier' and any tag value (this indicates changes of imported Danish address nodes).

	"post_labels": [
	    {"regex": [{".changes.tag.osak:identifier": ""}], "label": "address-node-change"}
	],

Format of regex on changes are:

        .changes[.action][.element-type].elements

where optional '.action' is either '.modify', '.create' or '.delete' and optional '.element-type' is either '.node', '.way' or '.relation'. The address change example above does not specify any action or element type i.e. any change of any element type will match.

Architecture

The general architecture is shown below. See also the Kubernetes architecture.

Image

Components

  • osmtracker.py The core functionality with multiple roles depending on arguments.
  • apiserver.py An API server providing access to the database. The OpenAPI/Swagger API specicication is in apispec.yaml.
  • elastic-gw.py An ElasticSearch sync componentat which pushes changesets to ElasticSearch.
  • db.py MongoDB backend for storing changeset information
  • messagebus.py Am AMQP (RabbitMQ) messagebus for interconnection components.
  • osm/changeset.py The class which contain the main analysis code.

The osmtracker.py script tracks OpenStreetMap minutely diffs and optionally filters them through a bounding-box polygon (country-based polygons can be found on http://download.geofabrik.de/). Changesets found to be within the area of interest are analysed further and a number of backends generate output. The HTML backends provide HTML data which can be served through a web-server. The client-side parts include a javascript-based poll feature that will update the changeset list whenever it changes (special care has been taken to minimize network load).

Configuration is provided through the config.json file.

Dependencies

See requirements.txt and Dockerfiles.

Links