Skip to content
A set of tools and documentation meant to help processing Naturvårdsverket's Nationella Marktäckedata (NMD) of Sweden
Python Shell
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data
old
translations
.gitignore
LICENSE
README.md
align-border-nodes.py
backport-tags.py
common.py
conflate-nodes.py
conflate-places.py
conflate-pways.py
do.py
filter-osm.py
geofabrik_to_places.sh
grass-pipeline.sh
match-features.py
negate-raster.py
nmd-gml-to-osm.py
plow-roads.py
positive-ids.sh
print-osm-stats.py
remap-raster.py
sew-seams.py
shp_to_osm.sh
split-osm-by-limit.py
water.py

README.md

nmd-osm-tools

This project is work in progress as of August 2019.

A set of tools and documentation meant to help processing Naturvårdsverket's Nationella Marktäckedata (NMD) of Sweden.

  • do.py - a top-level script to generate an OSM XML file for a single tile for given kommun and tile coordinates.
  • nmd-gml-to-osm.py - convert a GML file to OSM file applying tagging scheme from common.py. Note that this is not a general purpose GML to OSM XML conversion script. Try this script instead if you need such a functionality: https://github.com/andrewguertin/ogr2osm
  • remap-raster.py - simplify import GeoTIFF by replacing some values with other
  • plow-roads.py - remove nodes that are too close to roads and apply last minute transformations to vector data.
  • match-features.py - for selected types of features find pairs that are close enough to be possible aliases and deprioritize them in the new data.

Other scripts may also be present, read their comments to understand when they might become useful.

Older scripts, no longer needed for the main data flow, are placed in old.

Please also note that these scripts are only tested and meant to work for a limited subset of features present in GML, OSM or other types of files produced for this project. They will likely fail for other valid data files using wider subset of OSM/GML features.

Revised data flow overview

Look at the do.py code to get an overview how an individual data tile is being generated.

The idea is to process both input data layer and alternative data layers, such as raster mask layer and vector roads and match layers, in parallel until tiles of identical sizes and positions are generated, and then combine data of all layers into the result.

TODO the diagram below is outdated.

+----------------------+       +-----------------------------+
|                      |       |                             |
| NMD-raster image     |       | Geofabrik export data in SHP|
|                      |       |                             |
+------------------+---+       +--------+--------------------+
                   |                    |
                   |                    |
                   |                    | gdal_rasterize
                   |                    |
                   |           +--------v-------------------+
                   |           |                            |
                   |           | OSM raster tile in TIFF    |
                   |           |                            |
                   |           +--------+-------------------+
                   |                    |
                   |                    | negate_raster.py
                   |                    |
                   |           +--------v--------------------+
                   |           |                             |
                   |           | Mask tile (empty/not empty) |
                   |           |                             |
                   |           +-+---------------------------+
                   |             |
                   v             v
                 gdal_vectorize -mask
                          +
                          |
                          |
                +---------v------------+
                |                      |
                | Vector data in GML   |
                |                      |
                +---------+------------+
                          |
                          | nmd-gml-to-osm.py
                          |
                +---------v------------+        +-------------------------------+
                |                      |        |                               |
                | Vector data in OSM   |        | JOSM loaded actual data layer |
                |                      |        |                               |
                +---------+------------+        +------------------+------------+
                          |                                        |
                          +-----> Open in JOSM, merge layers, <----+
                                  fix warnings and problems
                                            +
                                            v
                                    +-------+----+
                                    |            |
                                    | Changeset  |
                                    |            |
                                    +------------+

Details of GML to OSM conversion

       sverige.tif         sverige-mask.tif
            +                     |
            |                     |
            v                     v
make-kommun-tiles.py       make-kommun-tiles.py
           |                      +
           |                      |
           v                      v
      tiles.tif               tile|mask.tif
            +                     |
            +----------+     +----+
                       v     v
                gdal-polygonize.py
                v.in.org
                v.generalize filter=chaiken threshold =20
                v.generalize filter=douglas threshold =0.00005
                v.out.ogr
                nmd-gml-to-osm.py
                plow-roads.py
                filter-osm threshold = 12
                    +
                    |
                    v
                load into JOSM
                    +
                    |
                    v
                fix warnings
                    +
                    v                   existing OSM data layer
               merge layers <-------------+
                    +
                    |
                    |
                    v
                fix warnings after merge
                    +
                    |
                    |
                    v
                  upload

Notes on how to produce a mask layer

TODO finish me

  • Download country shape files from https://download.geofabrik.de

  • Hints on editing a OSM shape file in QGIS as the process is not user-friendly: https://mangomap.com/industries/web-mapping/tutorials/remove-unwanted-regions-from-map-data.html

  • Open the following layers in QGIS: gis_osm_buildings_a_free_1.shp, gis_osm_landuse_a_free_1.shp, gis_osm_roads_free_1.shp

  • For roads, create a buffer area layer with 0.00006 degrees gap. TODO: decide whether to include paths and smallest roads (e.g. tracks) to the mask layer.

  • TODO For houses, buffer them with 0.00006 degrees to prevent landuse creep around houses.

  • For landuse, select and remove objects with fclass "military" and "nature_reserve"

  • Merge three shape files: buffered roads, buildings and modified landuse into a single shape files

  • TODO include in the mask layer features for golf course, cemetery, football etc.

  • To convert the shape file into a positive raster mask: gdal_rasterize -ot Byte -a code -tr 0.000114489630537 0.000114489630537 -co COMPRESS=LZW -l mask-landuse-buildings sweden-latest-shp/mask-landuse-buildings.shp sweden-mask-new-3-wgs84.tif

  • To invert bit values of the raster mask (takes up to 30 GB of RAM to finish!) ~/tmp2/nmd-osm-tools/negate-raster.py sweden-mask-new-3-wgs84.tif sweden-mask-new-negate-wgs84.tif

  • Ignore this warning: ERROR 1: TIFFSetField:sweden-mask-new-negate-wgs84.tif: Cannot modify tag "PhotometricInterpretation" while writing

References

You can’t perform that action at this time.