Skip to content

Home

Jump to bottom Edit New page
A Holt edited this page Jan 2, 2021 · 118 revisions

Recipe for Generating Map Regions

(For a General description see https://openmaptiles.org/docs/generate/create-custom-extract/ -- earlier/underlying/related work at https://github.com/openmaptiles/extracts and https://openmaptiles.org/osm2vectortiles/)

  1. Start with Ubuntu 20.04.1+, 400+ GB of SSD space, and 8+ GB of RAM.

  2. Find a recent OSM vector rendition of the planet, in the correct .mbtiles format. For example, there's a 83.4GB version from 2020-11-19 in https://archive.org/download/osm-vector-mbtiles, that can takes several days to download (Step 6. below).

  3. Clone http://github.com/iiab/maps, https://github.com/iiab/maptools, and https://github.com/georgejhunt/extracts into /opt/iiab/, as follows: (noting the 'extracts' repo MUST use the 'iiab' branch!)

    sudo su -    # Run everything as root
mkdir /opt/iiab
cd /opt/iiab
git clone https://github.com/iiab/maps
git clone https://github.com/iiab/maptools
git clone https://github.com/georgejhunt/extracts -b iiab    # Or, follow up with "git checkout -b iiab"

  4. A bounding box can be selected by navigating to https://www.openstreetmap.org/export. But Far Easier: use https://boundingbox.klokantech.com to generate the 4 latitude/longitude numbers you need {West,South,East,North} a.k.a. {W,S,E,N} as a CSV, i.e. comma-separated values — by changing the "Drag & Drop" drop-down from "MARC" to "CSV RAW" (so you can cut & paste those 4 values, for Step 8. below).

  5. Install the Node.js packages (https://www.npmjs.com/package/@mapbox/tilelive and https://www.npmjs.com/package/node-mbtiles) as follows:

    cd /opt/iiab/extracts
npm install
# Line below not nec, as /opt/iiab/extracts/packages.json (with "npm install") takes care of this!?
# npm i -g @mapbox/tilelive @mapbox/mbtiles

  6. Download the (giant) aforementioned OSM vector tiles file, for the whole planet: (this will be the source data, for every .mbtiles region you extract)

    cd /opt/iiab/extracts/data
wget https://archive.org/download/osm-vector-mbtiles/2020-10-planet-14.mbtiles

  7. Download IIAB's current map-catalog.json: 

    wget http://d.iiab.io/content/OSM/vector-tiles/map-catalog.json

  8. Edit your interim/local copy of map-catalog.json to reflect the new region(s), adding or modifying any 20-line section(s) as necessary (that you will be extracting). For now, only 1 + 4 = 5 of the 20 lines matter: the region's name (should start with "osm-" or "sat-") and its 4 metadata fields indicating the bounding box's lat/long numbers {W,S,E,N}.

    nano map-catalog.json

  9. Create a new CSV (comma-separated values) spec file: (use map-catalog.json as input)

    ../../maptools/mkcsv.py    # Writes CSV to screen, for a quick look
../../maptools/mkcsv.py > iiab.csv    # Creates input file for ../create-extracts.sh
    • iiab.csv will be the input for create-extracts.sh (Step 11. below).
    • The format of iiab.csv can be understood by looking at legacy file extracts.csv — of course we generally don't put anything in the "country" and "city" fields when generating continent-like extracts.

  10. REVIEW:

    • The overall tool chain / data flowchart is: determine {W,S,E,N} bounding box -nano-> map-catalog.json -mkcsv.py-> iiab.csv -create-extracts.sh-> desired regional extract .mbtiles file(s) !
    • The programs in the maptools repo (mostly) take input from the current directory, and write to stdout (the screen) as you can see with mkcsv.py above. This allows quick viewing of sample output, and keeps you involved in naming your own output files.

  11. In a moment you will run create-extracts.sh, which in turn runs create_extracts.py, which repeatedly calls tilelive-copy to create the individual regional subset files. (This will be the slowest step, as Node.js etc can take overnight!)

    But first: modify /opt/iiab/extracts/create-extracts.sh and /opt/iiab/extracts/create_extracts.py to specify input and output filenames (just as an example, osm_haiti_z11-z14_2020.mbtiles ?) Once everything looks good, run it all: (which could take overnight!)

    ../create-extracts.sh

  12. OPTIONALLY: for regions that are more complex than a simple "rectangle" (e.g. "Spanish-Speaking Regions") multiple .mbtiles files can be combined into a single such file, using this bash script: merge_regions

  13. Finally, to help you clean up and prepare to publish, these can safely be executed to inspect their respective outputs:

    ../../maptools/make_bboxes.py
../../maptools/check_catalog.py
    • make_bboxes.py generates a GeoJSON file enumerating all regions' bounding boxes or perimeters, so that IIAB can display these during install and/or post-install, during everyday usage.
    • check_catalog.py now updates the values of some fields, and can be modified to make any changes to the map-catalog.json that may be desired. In other words, check_catalog.py helps auto-generate some of the ~20 fields needed by each region in map-catalog.json, so this can be publicly published (see the explanations to the right of the ~20 metadata fields at https://github.com/iiab/maps/wiki/Map-Catalog-Format-and-Definitions !)

  14. Publish/Upload all the resulting files (each .mbtiles, the overall map-catalog.json, and any others?!)

Add a custom footer
Add a custom sidebar
Clone this wiki locally