Skip to content
A simple program for converting OSM map data to the MapsForge format
PLpgSQL Python HTML Shell Batchfile
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin
custom_polys
lists
openandromaps
pymapsforge
.gitignore
LICENSE-APACHE-2.0.txt
LICENSE-CC-BY-SA-3.0.html
README.md
azure_notes.md
custom_polys_and_extracts.md
download_maps.py
elevation.md
poi-mapping.xml
process_maps.py
sea.osm
shape2osm.py

README.md

Overview

This project contains useful info/tools for processing OSM maps into the mapwriter file format (.map). It was developed to help generating offline map files for Locus on Android.

Dependencies / Prereqs

To run this program you'll need the following

  • A Linux environment
  • Python 3.x installed
  • Java 1.8 or higher installed (OpenJDK works)
  • wget installed
  • gdal-bin
  • python3-gdal (Ubuntu) installed

Installation

Head over to releases tab and download the latest. Extract the file and you're good to go. Everything is self-contained minus the above dependencies.

Running

To run the program, cd to the directory where you extracted the release and run './process_maps.py' with at least one '--map-list /path/to/list.txt parameter'. See the lists directory for examples on how to format the map list files.

Program Usage

usage: process_maps.py [-h] --map-list MAP_LIST [--use-ram]
                       [--max-heap-space MAX_HEAP_SPACE]
                       [--output-map-name OUTPUT_MAP_NAME] --maps-dir MAPS_DIR
                       [--no-map-download] [--tag-mapping TAG_MAPPING]
                       [--tag-transform TAG_TRANSFORM]

optional arguments:
  -h, --help            show this help message and exit
  --map-list MAP_LIST   a text file with one map URL per line, can be
                        specified more than once
  --use-ram             use RAM for mapsforge processing -- WARNING mapsforge
                        uses 10x the map size in RAM for processing (ie. 100Mb
                        map = 1Gb RAM usage), you want a LOT of RAM for this
                        option
  --max-heap-space MAX_HEAP_SPACE
                        set the max heap space for the JVM, use standard -Xmx
                        values, default (1g) should be fine if not using
                        --use-ram argument
  --output-map-name OUTPUT_MAP_NAME
                        set the output .map and .poi file names
  --maps-dir MAPS_DIR   Where downloaded maps will be stored/read from
  --no-map-download     Do NOT download maps, re-use maps from maps-dir
  --tag-mapping TAG_MAPPING
                        Specify a custom tag mapping xml file for use with
                        mapsforge processing
  --tag-transform TAG_TRANSFORM
                        Specify a tag transform file for use PRIOR to
                        mapsforge processing (this is DIFFERENT than the
                        mapsforge mapping xml file)

Examples

./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name nicaragua-costa-rica-panama --map-list lists/nicaragua-costa-rica-panama
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name au-east --map-list lists/au-east.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 16g --maps-dir cache --no-map-download --output-map-name pennsylvania --map-list lists/pennsylvania.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name idaho --map-list lists/idaho.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name australia --map-list lists/australia.txt
./process_maps.py --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name canada_ontario --map-list lists/ontario.txt
./process_maps.py --tag-transform ./openandromaps/tag-transform.xml --tag-mapping ./openandromaps/tag-mapping.min.xml --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name midwest --map-list lists/midwest.txt --map-list lists/ontario.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name usa_northeast --map-list lists/usa_northeast.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name usa_pacific --map-list lists/usa_pacific.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name usa_south --map-list lists/usa_south.txt
./process_maps.py --use-ram --max-heap-space 16g --maps-dir cache --no-map-download --output-map-name mexico --map-list lists/mexico.txt
./process_maps.py --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name mexico_central_america --map-list lists/mexico.txt --map-list lists/central_america.txt
./process_maps.py --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name central_america --map-list lists/central_america.txt
./process_maps.py --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name south_america --map-list lists/south_america.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name netherlands --map-list lists/netherlands.txt
./process_maps.py --use-ram --max-heap-space 24g --maps-dir cache --no-map-download --output-map-name spain --map-list lists/spain.txt
./process_maps.py --max-heap-space 8g --maps-dir cache --no-map-download --output-map-name france --map-list lists/france.txt
./process_maps.py --use-ram --max-heap-space 16g --maps-dir cache --no-map-download --output-map-name belgium --map-list lists/belgium.txt
./process_maps.py --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name luxembourg --map-list lists/luxembourg.txt
./process_maps.py --use-ram --max-heap-space 12g --maps-dir cache --no-map-download --output-map-name portugal --map-list lists/portugal.txt

Caching maps for re-use / re-processing

Included in the repo there is a download_maps.py script that is meant to help with downloading maps on a schedule or for caching. It uses the same logic as the main script but for download only.

Program usage

usage: download_maps.py [-h] --map-list MAP_LIST --map-dir MAP_DIR

optional arguments:
  -h, --help           show this help message and exit
  --map-list MAP_LIST  a text file with one map URL per line, can be specified
                       more than once
  --map-dir MAP_DIR    The directory where maps should be downloaded

Example Usage

./download_maps.py --map-dir ./cache \
    --map-list lists/central_america.txt

./download_maps.py --map-dir ./cache \
    --map-list lists/midwest.txt

./download_maps.py --map-dir ./cache \
    --map-list lists/south_america.txt

Performance

This deserves an expansion but for now: notes on how to improve performance on 'small' systems that may not have >64Gb of RAM to work with for larger maps.

  • tmpfs works well for source/tmp/output in disk mode for speedups
  • zram /should/ work well if disk mode eats a little too much RAM when backed by tmpfs
  • type=ram uses closer to 15-20x source map size in practice
  • stacking maps from multiple sources increases memory footprint beyond the rough rules of thumb that are called out by the mapsforge devs
  • threads=n>1 is a fast way to ruin a heap space (don't do this with type=ram as a general rule of thumb)
  • there are choke points that are not multi-threaded once the main osmosis 'bits' are done running, deal

Look Into

If you're concerned about Java heap usage, the ZGC announced at the 2018 fosdem conference may be useful. I have no had success building a jre/jdk from sources but you may find the following link useful.

Licencing

All code is licensed Apache 2.0 and all non-code is licensed Creative Commons CC-BY-SA-3.0

You can’t perform that action at this time.