Hydrological tools for OpenStreetMap data
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
examples
html
notebooks
osm_dq
sample_data
LICENSE
README.md
__init__.py
osmconf_osm2dh.ini
run_osm_dq.py

README.md

hydro-osm

Hydrological tools for OpenStreetMap data

Hydro-osm is intended to become a toolbox to convert OpenStreetMap data into data layers that can be readily used for hydrological and hydraulic modelling. At the moment, the tools provide step number 1: data quality checking in particular for hydraulic modelling and the generation of a topologically correct 1D network.

It provides three quality checks

  • checking of the tagging data model
  • checking of the connectivity of the network, identifying disconnected parts in the network
  • checking crossings of waterways and roads

General run time information

the data quality routines are run through the python code run_osm_dq.py (see run_osm_dq.sh). The following code brings up a help screen

python run_osm_dq.py -h

Usage: run_osm_dq.py [options]

Options:
  -h, --help            show this help message and exit
  -q, --quiet           do not print status messages to stdout
  -o, --osm_download    retrieve osm data over bounding box and store in
                        osm_file in .ini file
  -i INIFILE, --ini=INIFILE
                        ini configuration file
  -c CHECK, --check=CHECK
                        which check to perform, can be: data_model,
                        connectivity, crossings
  -d DEST_PATH, --destination=DEST_PATH
                        Destination folder for reporting
  -p PREFIX, --prefix=PREFIX
                        Prefix for reporting files

A typical short written usage of the quality checks is as follows:

python run_osm_dq.py -i <INIFILE> -c <CHECK> -d <DEST_PATH> -p <PREFIX>

Structure of .ini file

The ini file contains all settings needed to perform the quality checks. For the different quality checks, different ini files may be required. The typical sections found are:

[input_data]: here the user can insert the osm file that needs be checked (osm_file), an OpenDataKit (ODK) xml-formatted configuration file (see examples) the extent (xmin, xmax, ymin, ymax) of the target area, the layer index (int) that should be used (layer_index) and the layer type that we'd be looking at (layer_type). The ODK file can be used to ingest a followed data model. This saves you a lot of duplication of configuration of the data model within your ini file, and ensures that the QA procedures follow exactly the data model that was also used during collection of the data with OpenDataKit. As ODK forms allow for a conditional data model, also conditional checks will be performed. For instance if shape of a drain is circular, then also check if a diameter is present. But if the shape is rectangular, then you don't need a radius, instead you need a width and depth.

[bounds]: if provided, it defines a filter for polygons, that define user-specified geographical regions for which the check should be performed. A layer_index should be selected (when the file contains multiple layers). Care should be taken that a Polygon layer is selected. For .osm files, the Polygon layer typically is layer number 3. Furthermore a key and a comma-separated list of values, or single value should be provided that filter out specific Polygons. In the examples, we use the key "name" and as value a list of neighbourhood names, stored under the osm tag "name".

[filter]: here it is defined which key/value pairs should be considered to filter out data that should be checked for quality. The user provides one key (in field key) and a comma-separated list of values (in field value).

[key_types]: if you use an ODK configuration file, then you don't need to consider this section at all, just leave it out! If not, then this section provides a set of keys for which the filtered elements should be checked for, along with the expected data type for this key. The entries are typically as follows:

key1 = datatype1
key2 = datatype2

etc...

where key1, key2 should be replaced by the actual name of the tag, and datatype1 and datatype2 should be replaced by the type of data, which can be str, int or float.

[key_add]: in case a file may contain features with more properties than included in the mandated data model, you can add an arbitrary number of properties here, along with their data type, very much in the same way as in the entry [key_types]. These properties will not be checked for consistency, but will simply be added to the output file. This is convenient in case that you wish the output file (including its data quality assurance flags) to be used for further analysis or uptake in e.g. OpenStreetMap.

[key_ranges]: the range of values for each key that is allowed. For floats this should be a comma-separated list of 2 values (minimum and maximum) while for integers and strings, this can be a comma-separated list of any number of allowed entries.

[connectivity]: only used when checking for connectivity. Here a filter for selected valid outflow points is provided as comma-separated list of integers, and a tolerance, which is a float, identifying any allowed snapping distance for non-connected elements. In case the user expects that all line segments filtered should be completely connected, this value should be set to zero. The unit of this option is the spatial unit belonging to the projection of the used input file. In most cases this will be WGS84 lat-lon.

[filter_highway]: only used with crossings check. Similar to [filter], but used to filter out roads (where [filter] is assumed to filter out waterways).

[filter_bridge]: only used with crossings check. Similar to [filter], but used to filter out bridges.

[filter_tunnel]: only used with crossings check. Similar to [filter], but used to filter out tunnels (or culverts).

Running data quality routines

Running the data quality routines should be done with the following command:

python run_osm_dq.py -i <INIFILE> -c <CHECK> -d <DEST_PATH> -p <PREFIX>

where an .ini file is selected with -i, a check is selected with -c, the destination path, where outputs are written is provided with -d, and a prefix to output file names (for later reference and easy recognition of files) can be provided with -p. The choices for data quality assurance checks are:

Data model (data_model)

Checks the completeness of the data_model for given filtered features. Features are selected geographically, using a user-specified set of polygons (geographically), as well as tag-specific, using a user-specified tag filter. For instance, in our examples, we use the section [bounds] to provide a set of geographical areas on which to perform the check, and we use the [filter] section to filter out features using tags. The check then uses either the data model located within the supplied ODK file, or the [key_types] and [key_values] sections to understand the data model and to check if the filtered features follow the data model.

Connectivity (connectivity)

Checks topological connectivity of a line network with respect to a (set of) user-selected end segments or points representing for instance stream beginnings or outlets. The end segments are filtered by the user in sections named [connectivity_XXXX] where XXXX can be filled in by the user. XXXX can for instance be begin or outlet. In each connectivity section, a user defines the following:

Key Value
file the vector file that should be used to filter out point or segments that represent connectivity check points. If this is not provided, this may be left open
key The tag that identifies the type of drainage connection features (e.g. feature_type)
value The (comma-separated value(s) of the tag, that identify the drainage connection features that you would like to check connectivity against (e.g. outflow, no_exit, other)
uniqueid A tag that usually is unique per connection feature

For each connectivity section, the connectivity check is performed separately. In this way you can check in one go for connectivity of outlets, as well as begin points, as well as potentially other points you expect connectivity against. You may for instance also add a check for connectivity to silt traps if you wish to understand which parts of drainage network have a connection to siltation facilities. This section you could call [connectivity_silt] for instance.

Crossings (crossings)

Three additional tag filters need to be provided for this, [filter_highway], [filter_bridge] and [filter_tunnel]. The check will use the filter [filter] to filter out water ways, [filter_highway] to filter out roads, [filter_bridge] to filter out bridges and [filter_tunnel] to filter out tunnels such as culverts. The check then searches for crossing water way and road elements and will check if these crossing are either a bridge or a tunnel using the [filter_bridge] and [filter_tunnel] tags. It will then report the found crossings and what the type of crossing is (bridge, tunnel or undefined). The check uses the [bounds] filter to perform the check per Polygon. If this is not provided, the complete data set will be checked and reported as one region.

output of data quality routines

Data model

The data model check provides 2 outputs:

  • Excel file: it has different tabs for each region selected in the [bounds] section. Within each tab, per tag, a summary is provided of the amount of features that has an appropriate value for the tag (0), the amount that has an invalid value (1), the amount that has a value but of the wrong datatype (2) and the amount, not having a value at all.
  • GeoJSON file: this file can be loaded in GIS software such as QGIS and shows all filtered features. Each feature contains the tags that were relevant in the selected data model, as well as new tags, similarly names, but with a suffix '_flag' that show the result of the check on that feature and tag. For instance if a check is performed on the tag 'width', the check will return the original value of the tag 'width' for each feature, but also a tag 'width_flag'. This tag can have the value 0 (valid value), 1 (invalid value), 2 (incorrect data type) or 3 (no value). It is then straightforward to use these flag values to color code the GIS layout. Any additional tags, defined in the section [key_add] are simply copied to the GeoJSON without any checks. [key_add] can help you to provide a complete file, ready for upload to OpenStreetMap. One unique additional key is added called geom_check. This field is 0 when the feature contains a valid geometry and 1 when the geometry is in any way invalid.

Connectivity

This check is performed either separately or combined with the data model check. It is automatically executed after the data model quality check, if any [connectivity_XXXX] section is provided in the .ini file (where you can select your own naming for the part XXX). For each [connectivity_XXXX] section in the .ini file, a tag is added to the output GeoJSON, called connectivity_XXXX with values being zero if the feature is not connected to any points filtered out with the settings under the respective filter, or the value of the uniqueid of the connection feature it is connected to.

Crossings

The check gives two outputs:

  • Excel file: the excel shows (per selected region, using the [bounds] section) the amount of crossings with a bridge, tunnel or undefined crossing.
  • GeoJSON files. 3 files are provided. 2 files contain the filtered waterways and roads respectively. The last file is a Point file containing the crossings found. Each crossing has a set of tags, being the name of the filtered region in which the crossing is located, the osm_id of the crossing waterway, the osm_id of the crossing highway, a flag value (0 when a known crossing is found, 1 when no crossing structure is found).

Examples

The examples folder contains a set of .ini files as well as .bat files. The .bat files can be used to run the examples. Sample data to run the examples with, is provided in the folder sample_data. Please unzip the file sample_data\kigogo.zip to the folder examples before running the examples.