Custom PBF parsing #82

msbarry · 2022-02-17T11:11:20Z

Previously planetiler depended on graphhopper utilities for PBF parsing. This change pulls in PBF parsing so we can have more control over the threading model, performance optimizations, and eventually support for locations on ways.

Left to do:

move header parsing into PBF parser/decoder
tests for PBF parser/decoder
comments/cleanup
check for ascending element IDs, nodes before ways, ways before rels
performance/low memory test

Then after this change:

remove graphhopper dependency and update HPPC to the latest version
try to parallelize element handling in pass1
mmap improvements: use bigger mmap segments, madvise random

github-actions · 2022-02-17T11:49:39Z

Base 4ad4d95 This Branch 4d32b2e

0:01:44 DEB [mbtiles] - Tile stats:
0:01:44 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:44 DEB [mbtiles] - z1 avg:4k max:4k
0:01:44 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:44 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:44 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:44 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:44 DEB [mbtiles] - z6 avg:1.4k max:24k
0:01:44 DEB [mbtiles] - z7 avg:896 max:33k
0:01:44 DEB [mbtiles] - z8 avg:365 max:48k
0:01:44 DEB [mbtiles] - z9 avg:295 max:278k
0:01:44 DEB [mbtiles] - z10 avg:164 max:232k
0:01:44 DEB [mbtiles] - z11 avg:107 max:131k
0:01:44 DEB [mbtiles] - z12 avg:85 max:118k
0:01:44 DEB [mbtiles] - z13 avg:72 max:109k
0:01:44 DEB [mbtiles] - z14 avg:68 max:256k
0:01:44 DEB [mbtiles] - all avg:70 max:0
0:01:44 DEB [mbtiles] -  # features: 5,275,801
0:01:44 DEB [mbtiles] -     # tiles: 4,115,465
0:01:44 INF [mbtiles] - Finished in 25s cpu:45s gc:1s avg:1.8
0:01:44 INF [mbtiles] -   reader   1x(8% cpu:2s wait:20s)
0:01:44 INF [mbtiles] -   encoder  2x(44% cpu:11s wait:6s)
0:01:44 INF [mbtiles] -   writer   1x(37% cpu:9s wait:13s)
0:01:44 INF - Finished in 1m44s cpu:3m5s gc:4s avg:1.8
0:01:44 INF - FINISHED!

0:01:44 INF - ----------------------------------------
0:01:44 INF - 	overall          1m44s cpu:3m5s gc:4s avg:1.8
0:01:44 INF - 	lake_centerlines 0.6s cpu:1s gc:0s avg:1.7
0:01:44 INF - 	  read     1x(50% cpu:0.3s)
0:01:44 INF - 	  process  2x(10% cpu:0.1s)
0:01:44 INF - 	  write    1x(0% cpu:0s)
0:01:44 INF - 	water_polygons   25s cpu:45s gc:2s avg:1.8
0:01:44 INF - 	  read     1x(61% cpu:15s sys:1s wait:2s)
0:01:44 INF - 	  process  2x(28% cpu:7s wait:12s)
0:01:44 INF - 	  write    1x(4% cpu:1s wait:24s)
0:01:44 INF - 	natural_earth    10s cpu:15s gc:0.1s avg:1.6
0:01:44 INF - 	  read     1x(84% cpu:8s sys:1s)
0:01:44 INF - 	  process  2x(19% cpu:2s wait:8s)
0:01:44 INF - 	  write    1x(0% cpu:0s wait:9s)
0:01:44 INF - 	osm_pass1        4s cpu:7s gc:0s avg:1.9
0:01:44 INF - 	  pbf      1x(8% cpu:0.3s wait:3s)
0:01:44 INF - 	  process  1x(38% cpu:1s wait:2s)
0:01:44 INF - 	osm_pass2        28s cpu:56s gc:0.3s avg:2
0:01:44 INF - 	  pbf      1x(1% cpu:0.3s wait:17s done:11s)
0:01:44 INF - 	  process  2x(73% cpu:21s)
0:01:44 INF - 	  write    1x(1% cpu:0.3s wait:28s)
0:01:44 INF - 	boundaries       0s cpu:0.1s gc:0s avg:1.6
0:01:44 INF - 	sort             4s cpu:5s gc:0s avg:1.3
0:01:44 INF - 	  worker  2x(43% cpu:2s done:2s)
0:01:44 INF - 	mbtiles          25s cpu:45s gc:1s avg:1.8
0:01:44 INF - 	  reader   1x(8% cpu:2s wait:20s)
0:01:44 INF - 	  encoder  2x(44% cpu:11s wait:6s)
0:01:44 INF - 	  writer   1x(37% cpu:9s wait:13s)
0:01:44 INF - ----------------------------------------
0:01:44 INF - 	features	268MB
0:01:44 INF - 	mbtiles	514MB
-rw-r--r-- 1 runner docker 53M Mar  1 01:36 run.jar

0:01:37 DEB [mbtiles] - Tile stats:
0:01:37 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:37 DEB [mbtiles] - z1 avg:4k max:4k
0:01:37 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:37 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:37 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:37 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:37 DEB [mbtiles] - z6 avg:1.4k max:24k
0:01:37 DEB [mbtiles] - z7 avg:896 max:33k
0:01:37 DEB [mbtiles] - z8 avg:365 max:48k
0:01:37 DEB [mbtiles] - z9 avg:295 max:279k
0:01:37 DEB [mbtiles] - z10 avg:164 max:232k
0:01:37 DEB [mbtiles] - z11 avg:107 max:131k
0:01:37 DEB [mbtiles] - z12 avg:85 max:118k
0:01:37 DEB [mbtiles] - z13 avg:72 max:109k
0:01:37 DEB [mbtiles] - z14 avg:68 max:257k
0:01:37 DEB [mbtiles] - all avg:70 max:0
0:01:37 DEB [mbtiles] -  # features: 5,275,803
0:01:37 DEB [mbtiles] -     # tiles: 4,115,465
0:01:37 INF [mbtiles] - Finished in 25s cpu:45s avg:1.8
0:01:37 INF [mbtiles] -   read    1x(8% 2s wait:20s)
0:01:37 INF [mbtiles] -   encode  2x(44% 11s wait:6s)
0:01:37 INF [mbtiles] -   write   1x(37% 9s wait:13s)
0:01:37 INF - Finished in 1m37s cpu:2m47s gc:4s avg:1.7
0:01:37 INF - FINISHED!

0:01:37 INF - ----------------------------------------
0:01:37 INF - 	overall          1m37s cpu:2m47s gc:4s avg:1.7
0:01:37 INF - 	lake_centerlines 0.5s cpu:0.8s avg:1.7
0:01:37 INF - 	water_polygons   21s cpu:38s gc:2s avg:1.8
0:01:37 INF - 	  read     1x(60% 13s wait:2s)
0:01:37 INF - 	  process  2x(26% 6s wait:10s)
0:01:37 INF - 	  write    1x(3% 0.7s wait:20s)
0:01:37 INF - 	natural_earth    8s cpu:13s avg:1.6
0:01:37 INF - 	  read     1x(86% 7s sys:1s)
0:01:37 INF - 	  process  2x(19% 2s wait:7s)
0:01:37 INF - 	  write    1x(0% 0s wait:7s)
0:01:37 INF - 	osm_pass1        3s cpu:6s avg:1.8
0:01:37 INF - 	  read     1x(3% 0.1s wait:3s)
0:01:37 INF - 	  parse    1x(67% 2s)
0:01:37 INF - 	  process  1x(34% 1s wait:1s)
0:01:37 INF - 	osm_pass2        26s cpu:51s avg:2
0:01:37 INF - 	  read     1x(0% 0s wait:3s done:23s)
0:01:37 INF - 	  process  2x(75% 19s)
0:01:37 INF - 	  write    1x(1% 0.3s wait:25s)
0:01:37 INF - 	boundaries       0s cpu:0.1s avg:1.7
0:01:37 INF - 	sort             4s cpu:5s avg:1.2
0:01:37 INF - 	  worker  2x(43% 2s done:2s)
0:01:37 INF - 	mbtiles          25s cpu:45s avg:1.8
0:01:37 INF - 	  read    1x(8% 2s wait:20s)
0:01:37 INF - 	  encode  2x(44% 11s wait:6s)
0:01:37 INF - 	  write   1x(37% 9s wait:13s)
0:01:37 INF - ----------------------------------------
0:01:37 INF - 	features	268MB
0:01:37 INF - 	mbtiles	514MB
-rw-r--r-- 1 runner docker 53M Mar  1 01:34 run.jar

https://github.com/onthegomap/planetiler/actions/runs/1913591917

ℹ️ Base Logs 4ad4d95

0:00:00 DEB - argument: config=null (path to config file)
0:00:00 DEB - argument: area=rhode island (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet'))
0:00:00 INF - Using in-memory stats

0:00:00 INF [overall] - Starting...
0:00:00 DEB - argument: bounds=Env[-74.07 : -17.84, 21.34 : 43.55] (bounds)
0:00:00 DEB - argument: threads=2 (num threads)
0:00:00 DEB - argument: loginterval=10 seconds (time between logs)
0:00:00 DEB - argument: minzoom=0 (minimum zoom level)
0:00:00 DEB - argument: maxzoom=14 (maximum zoom level (limit 14))
0:00:00 DEB - argument: defer_mbtiles_index_creation=false (skip adding index to mbtiles file)
0:00:00 DEB - argument: optimize_db=false (optimize mbtiles after writing)
0:00:00 DEB - argument: emit_tiles_in_order=true (emit tiles in index order)
0:00:00 DEB - argument: force=false (force overwriting output file)
0:00:00 DEB - argument: gzip_temp=false (gzip temporary feature storage (uses more CPU, but less disk space))
0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks)
0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks)
0:00:00 DEB - argument: nodemap_type=sortedtable (type of node location map: noop, sortedtable, or sparsearray)
0:00:00 DEB - argument: nodemap_storage=mmap (storage for location map: mmap or ram)
0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP)
0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP)
0:00:00 DEB - argument: download_chunk_size_mb=100 (Size of file chunks to download in parallel in megabytes)
0:00:00 DEB - argument: download_threads=1 (Number of parallel threads to use when downloading each file)
0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level)
0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level)
0:00:00 DEB - argument: tmpdir=data/tmp (temp directory)
0:00:00 DEB - argument: only_download=false (download source data then exit)
0:00:00 DEB - argument: download=false (download sources)
0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit)
0:00:00 DEB - argument: fetch_wikidata=false (fetch wikidata translations then continue)
0:00:00 DEB - argument: use_wikidata=true (use wikidata translations)
0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file)
0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path)
0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path)
0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path)
0:00:00 DEB - argument: osm_path=data/sources/rhode_island.osm.pbf (osm OSM input file path)
0:00:00 DEB - argument: mbtiles=data/out.mbtiles (mbtiles output file)
0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names)
0:00:00 DEB - argument: languages=am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh (languages to use)
0:00:00 DEB - argument: only_layers= (Include only certain layers)
0:00:00 DEB - argument: exclude_layers= (Exclude certain layers)
0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries)
0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13)
0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13)
0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways)
0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield))
0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways)
0:00:00 DEB - argument: mbtiles_name=OpenMapTiles ('name' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a> ('attribution' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_version=3.13.0 ('version' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_type=baselayer ('type' attribute for mbtiles metadata)
0:00:00 DEB - argument: help=false (show arguments then exit)
0:00:00 INF - Building BasemapProfile profile into data/out.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   mbtiles: Encode each tile and write to data/out.mbtiles
0:00:00 INF - error loading /home/runner/work/planetiler/planetiler/data/sources/wikidata_names.json: java.nio.file.NoSuchFileException: data/sources/wikidata_names.json
0:00:00 INF - Using merge sort feature map, chunk size=1000mb workers=2
0:00:01 INF - dataFileCache open start

0:00:02 INF [lake_centerlines] - Starting...
0:00:02 INF [lake_centerlines] -  read: [  14k 100%  29k/s ] write: [    0    0/s ] 0    
    cpus: 1.7 gc:  0% mem: 165M/4.2G postGC: 45M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:02 INF [lake_centerlines] - Finished in 0.6s cpu:1s gc:0s avg:1.7
0:00:02 INF [lake_centerlines] -   read     1x(50% cpu:0.3s)
0:00:02 INF [lake_centerlines] -   process  2x(10% cpu:0.1s)
0:00:02 INF [lake_centerlines] -   write    1x(0% cpu:0s)

0:00:02 INF [water_polygons] - Starting...
0:00:12 INF [water_polygons] -  read: [   2k  14%  204/s ] write: [  47k 4.7k/s ] 3.2M 
    cpus: 1.9 gc: 12% mem: 1.3G/4.2G postGC: 1.1G
    read(57%) ->    (0/1k) -> process(43% 15%) -> (679/53k) -> write( 0%)
0:00:22 INF [water_polygons] -  read: [ 5.3k  37%  327/s ] write: [ 298k  25k/s ] 15M  
    cpus: 1.7 gc:  6% mem: 2.8G/4.2G postGC: 1.4G
    read(72%) ->    (0/1k) -> process(28% 19%) -> (1.2k/53k) -> write( 1%)
0:00:28 INF [water_polygons] -  read: [  14k 100% 1.7k/s ] write: [ 4.3M 750k/s ] 186M 
    cpus: 1.7 gc:  7% mem: 2.2G/4.2G postGC: 1.7G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:28 INF [water_polygons] - Finished in 25s cpu:45s gc:2s avg:1.8
0:00:28 INF [water_polygons] -   read     1x(61% cpu:15s sys:1s wait:2s)
0:00:28 INF [water_polygons] -   process  2x(28% cpu:7s wait:12s)
0:00:28 INF [water_polygons] -   write    1x(4% cpu:1s wait:24s)
0:00:28 INF [natural_earth] - unzipping /home/runner/work/planetiler/planetiler/data/sources/natural_earth_vector.sqlite.zip to data/tmp/natearth.sqlite

0:00:33 INF [natural_earth] - Starting...
0:00:43 INF [natural_earth] -  read: [ 349k 100%  38k/s ] write: [  181   20/s ] 186M 
    cpus: 1.6 gc:  1% mem: 974M/4.2G postGC: 173M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:43 INF [natural_earth] - Finished in 10s cpu:15s gc:0.1s avg:1.6
0:00:43 INF [natural_earth] -   read     1x(84% cpu:8s sys:1s)
0:00:43 INF [natural_earth] -   process  2x(19% cpu:2s wait:8s)
0:00:43 INF [natural_earth] -   write    1x(0% cpu:0s wait:9s)

0:00:43 INF [osm_pass1] - Starting...
0:00:47 INF [osm_pass1] -  nodes: [ 4.5M 1.1M/s ] 353M  ways: [ 326k  85k/s ] rels: [ 7.3k 1.9k/s ]
    cpus: 1.9 gc:  0% mem: 1.9G/4.2G postGC: 683M hppc: 882k
    parse() -> pbf( -%) ->   (0/70k) -> process( -%)
0:00:47 DEB [osm_pass1] - nodes: 4,572,227 ways: 326,909 relations: 7,387
0:00:47 INF [osm_pass1] - Finished in 4s cpu:7s gc:0s avg:1.9
0:00:47 INF [osm_pass1] -   pbf      1x(8% cpu:0.3s wait:3s)
0:00:47 INF [osm_pass1] -   process  1x(38% cpu:1s wait:2s)

0:00:47 INF [osm_pass2] - Starting...
0:00:51 DEB [osm_pass2:process] - Sorting long long multimap...
0:00:51 DEB [osm_pass2:process] - Sorted long long multimap time:0.020331629s
0:00:51 WAR [osm_pass2:process] - No GB polygon for inferring route network types
0:00:57 INF [osm_pass2] -  nodes: [ 4.5M 100% 457k/s ] 353M  ways: [ 146k  45%  14k/s ] rels: [    0   0%    0/s ] features: [ 4.8M  55k/s ] 227M 
    cpus: 2 gc:  1% mem: 1.9G/4.2G postGC: 808M hppc: 1.6M
    parse(12%) -> pbf( 3%) -> (51k/53k) -> process(61% 56%) -> (1.1k/53k) -> write( 2%)
0:01:07 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 353M  ways: [ 326k 100%  18k/s ] rels: [   2k  28%  208/s ] features: [ 5.2M  37k/s ] 255M 
    cpus: 2 gc:  1% mem: 2.1G/4.2G postGC: 822M hppc:  29M
    parse( -%) -> pbf( -%) -> (5.2k/53k) -> process(77% 80%) -> (897/53k) -> write( 1%)
0:01:15 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 353M  ways: [ 326k 100%    0/s ] rels: [ 7.3k 100%  641/s ] features: [ 5.2M 3.2k/s ] 268M 
    cpus: 2 gc:  0% mem: 1.3G/4.2G postGC: 818M hppc:  29M
    parse( -%) -> pbf( -%) ->   (0/53k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:01:15 DEB [osm_pass2] - nodes: 4,572,227 ways: 326,909 relations: 7,387
0:01:15 INF [osm_pass2] - Finished in 28s cpu:56s gc:0.3s avg:2
0:01:15 INF [osm_pass2] -   pbf      1x(1% cpu:0.3s wait:17s done:11s)
0:01:15 INF [osm_pass2] -   process  2x(73% cpu:21s)
0:01:15 INF [osm_pass2] -   write    1x(1% cpu:0.3s wait:28s)

0:01:15 INF [boundaries] - Starting...
0:01:15 INF [boundaries] - Creating polygons for 1 boundaries
0:01:15 WAR [boundaries] - Unable to form closed polygon for OSM relation 148838 (likely missing edges)
0:01:15 INF [boundaries] - Finished creating 0 country polygons
0:01:15 INF [boundaries] - Finished in 0s cpu:0.1s gc:0s avg:1.6
0:01:15 INF - Deleting node.db to make room for mbtiles

0:01:15 INF [sort] - Starting...
0:01:19 INF [sort] -  chunks: [   1 /   1 100% ] 268M 
    cpus: 1.3 gc:  0% mem: 1.8G/4.2G postGC: 818M
    ->     (0/4) -> worker( -%  -%)
0:01:19 INF [sort] - Finished in 4s cpu:5s gc:0s avg:1.3
0:01:19 INF [sort] -   worker  2x(43% cpu:2s done:2s)
0:01:19 INF - read:1s write:1s sort:1s

0:01:19 INF [mbtiles] - Starting...
0:01:20 DEB [mbtiles:writer] - Execute mbtiles: create table metadata (name text, value text);
0:01:20 DEB [mbtiles:writer] - Execute mbtiles: create unique index name on metadata (name);
0:01:20 DEB [mbtiles:writer] - Execute mbtiles: create table tiles (zoom_level integer, tile_column integer, tile_row, tile_data blob);
0:01:20 DEB [mbtiles:writer] - Execute mbtiles: create unique index tile_index on tiles (zoom_level, tile_column, tile_row)
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: name=OpenMapTiles
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: format=pbf
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: version=3.13.0
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: type=baselayer
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: bounds=-74.07,21.34,-17.84,43.55
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: center=-45.955,32.445,3
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: minzoom=0
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: maxzoom=14
0:01:20 DEB [mbtiles:writer] - Set mbtiles metadata: json={"vector_layers":[{"id":"aerodrome_label","fields":{"name_int":"String","iata":"String","ele_ft":"Number","name_de":"String","name":"String","icao":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":10,"maxzoom":14},{"id":"aeroway","fields":{"ref":"String","class":"String"},"minzoom":10,"maxzoom":14},{"id":"boundary","fields":{"disputed":"Number","admin_level":"Number","maritime":"Number","disputed_name":"String"},"minzoom":0,"maxzoom":14},{"id":"building","fields":{"colour":"String","render_height":"Number","render_min_height":"Number"},"minzoom":13,"maxzoom":14},{"id":"housenumber","fields":{"housenumber":"String"},"minzoom":14,"maxzoom":14},{"id":"landcover","fields":{"subclass":"String","class":"String","_numpoints":"Number"},"minzoom":8,"maxzoom":14},{"id":"landuse","fields":{"class":"String"},"minzoom":4,"maxzoom":14},{"id":"mountain_peak","fields":{"name_int":"String","customary_ft":"Number","ele_ft":"Number","name_de":"String","name":"String","rank":"Number","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":7,"maxzoom":14},{"id":"park","fields":{"name_int":"String","name_de":"String","name":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"place","fields":{"name:fy":"String","name_int":"String","capital":"Number","name:uk":"String","name:pl":"String","name:nl":"String","name:ru":"String","name:be":"String","name:ko":"String","name_de":"String","name":"String","rank":"Number","name:en":"String","name:eo":"String","class":"String","name:ta":"String","name:hu":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":2,"maxzoom":14},{"id":"poi","fields":{"name_int":"String","level":"Number","name:nonlatin":"String","layer":"Number","name_de":"String","name":"String","subclass":"String","indoor":"Number","name:en":"String","class":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":12,"maxzoom":14},{"id":"transportation","fields":{"access":"String","brunnel":"String","expressway":"Number","surface":"String","bicycle":"String","level":"Number","ramp":"Number","mtb_scale":"String","toll":"Number","oneway":"Number","layer":"Number","network":"String","horse":"String","service":"String","subclass":"String","class":"String","foot":"String"},"minzoom":4,"maxzoom":14},{"id":"transportation_name","fields":{"name_int":"String","name:nonlatin":"String","route_4":"String","route_3":"String","route_2":"String","route_1":"String","layer":"Number","network":"String","ref":"String","name_de":"String","name":"String","subclass":"String","ref_length":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"water","fields":{"intermittent":"Number","class":"String"},"minzoom":0,"maxzoom":14},{"id":"water_name","fields":{"name_int":"String","name:nonlatin":"String","name_de":"String","name":"String","intermittent":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":9,"maxzoom":14},{"id":"waterway","fields":{"name_int":"String","brunnel":"String","name_de":"String","_relid":"Number","intermittent":"Number","name":"String","class":"String","name:latin":"String","name_en":"String"},"minzoom":4,"maxzoom":14}]}
0:01:20 INF [mbtiles:writer] - Starting z0
0:01:20 INF [mbtiles:writer] - Finished z0 in 0s cpu:0s gc:0s avg:0, now starting z1
0:01:20 INF [mbtiles:writer] - Finished z1 in 0s cpu:0s gc:0s avg:0, now starting z2
0:01:20 INF [mbtiles:writer] - Finished z2 in 0s cpu:0s gc:0s avg:0, now starting z3
0:01:20 INF [mbtiles:writer] - Finished z3 in 0s cpu:0s gc:0s avg:63.8, now starting z4
0:01:20 INF [mbtiles:writer] - Finished z4 in 0s cpu:0s gc:0s avg:0, now starting z5
0:01:20 INF [mbtiles:writer] - Finished z5 in 0s cpu:0s gc:0s avg:0, now starting z6
0:01:20 INF [mbtiles:writer] - Finished z6 in 0s cpu:0s gc:0s avg:0, now starting z7
0:01:20 INF [mbtiles:writer] - Finished z7 in 0s cpu:0.1s gc:0s avg:2.1, now starting z8
0:01:22 INF [mbtiles:writer] - Finished z8 in 1s cpu:3s gc:0.1s avg:2, now starting z9
0:01:22 INF [mbtiles:writer] - Finished z9 in 0.5s cpu:1s gc:0s avg:2, now starting z10
0:01:22 INF [mbtiles:writer] - Finished z10 in 0.2s cpu:0.5s gc:0s avg:2, now starting z11
0:01:23 INF [mbtiles:writer] - Finished z11 in 0.8s cpu:2s gc:0s avg:2, now starting z12
0:01:25 INF [mbtiles:writer] - Finished z12 in 2s cpu:3s gc:0s avg:2, now starting z13
0:01:30 INF [mbtiles] -  features: [ 706k  13%  67k/s ] 268M  tiles: [ 291k  27k/s ] 46M  
    cpus: 2 gc:  8% mem: 3.1G/4.2G postGC: 855M
    reader( 4%) -> (213/217) -> encoder(57% 55%) -> (215/216) -> writer( 8%)
    last tile: 13/2470/3048 (z13 4%) https://www.openstreetmap.org/#map=13/41.77131/-71.45508
0:01:36 INF [mbtiles:writer] - Finished z13 in 11s cpu:22s gc:0.3s avg:2, now starting z14
0:01:40 INF [mbtiles] -  features: [ 3.5M  67% 279k/s ] 268M  tiles: [ 2.1M 189k/s ] 288M 
    cpus: 1.8 gc:  2% mem: 2.3G/4.2G postGC: 862M
    reader( 8%) ->   (0/217) -> encoder(47% 49%) -> (213/216) -> writer(41%)
    last tile: 14/5794/7151 (z14 38%) https://www.openstreetmap.org/#map=14/22.28910/-52.69043
0:01:44 INF [mbtiles:writer] - Finished z14 in 8s cpu:12s gc:0.1s avg:1.5
0:01:44 DEB [mbtiles] - Tile stats:
0:01:44 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:44 DEB [mbtiles] - z1 avg:4k max:4k
0:01:44 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:44 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:44 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:44 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:44 DEB [mbtiles] - z6 avg:1.4k max:24k
0:01:44 DEB [mbtiles] - z7 avg:896 max:33k
0:01:44 DEB [mbtiles] - z8 avg:365 max:48k
0:01:44 DEB [mbtiles] - z9 avg:295 max:278k
0:01:44 DEB [mbtiles] - z10 avg:164 max:232k
0:01:44 DEB [mbtiles] - z11 avg:107 max:131k
0:01:44 DEB [mbtiles] - z12 avg:85 max:118k
0:01:44 DEB [mbtiles] - z13 avg:72 max:109k
0:01:44 DEB [mbtiles] - z14 avg:68 max:256k
0:01:44 DEB [mbtiles] - all avg:70 max:0
0:01:44 DEB [mbtiles] -  # features: 5,275,801
0:01:44 DEB [mbtiles] -     # tiles: 4,115,465
0:01:44 INF [mbtiles] - Finished in 25s cpu:45s gc:1s avg:1.8
0:01:44 INF [mbtiles] -   reader   1x(8% cpu:2s wait:20s)
0:01:44 INF [mbtiles] -   encoder  2x(44% cpu:11s wait:6s)
0:01:44 INF [mbtiles] -   writer   1x(37% cpu:9s wait:13s)
0:01:44 INF - Finished in 1m44s cpu:3m5s gc:4s avg:1.8
0:01:44 INF - FINISHED!

0:01:44 INF - ----------------------------------------
0:01:44 INF - 	overall          1m44s cpu:3m5s gc:4s avg:1.8
0:01:44 INF - 	lake_centerlines 0.6s cpu:1s gc:0s avg:1.7
0:01:44 INF - 	  read     1x(50% cpu:0.3s)
0:01:44 INF - 	  process  2x(10% cpu:0.1s)
0:01:44 INF - 	  write    1x(0% cpu:0s)
0:01:44 INF - 	water_polygons   25s cpu:45s gc:2s avg:1.8
0:01:44 INF - 	  read     1x(61% cpu:15s sys:1s wait:2s)
0:01:44 INF - 	  process  2x(28% cpu:7s wait:12s)
0:01:44 INF - 	  write    1x(4% cpu:1s wait:24s)
0:01:44 INF - 	natural_earth    10s cpu:15s gc:0.1s avg:1.6
0:01:44 INF - 	  read     1x(84% cpu:8s sys:1s)
0:01:44 INF - 	  process  2x(19% cpu:2s wait:8s)
0:01:44 INF - 	  write    1x(0% cpu:0s wait:9s)
0:01:44 INF - 	osm_pass1        4s cpu:7s gc:0s avg:1.9
0:01:44 INF - 	  pbf      1x(8% cpu:0.3s wait:3s)
0:01:44 INF - 	  process  1x(38% cpu:1s wait:2s)
0:01:44 INF - 	osm_pass2        28s cpu:56s gc:0.3s avg:2
0:01:44 INF - 	  pbf      1x(1% cpu:0.3s wait:17s done:11s)
0:01:44 INF - 	  process  2x(73% cpu:21s)
0:01:44 INF - 	  write    1x(1% cpu:0.3s wait:28s)
0:01:44 INF - 	boundaries       0s cpu:0.1s gc:0s avg:1.6
0:01:44 INF - 	sort             4s cpu:5s gc:0s avg:1.3
0:01:44 INF - 	  worker  2x(43% cpu:2s done:2s)
0:01:44 INF - 	mbtiles          25s cpu:45s gc:1s avg:1.8
0:01:44 INF - 	  reader   1x(8% cpu:2s wait:20s)
0:01:44 INF - 	  encoder  2x(44% cpu:11s wait:6s)
0:01:44 INF - 	  writer   1x(37% cpu:9s wait:13s)
0:01:44 INF - ----------------------------------------
0:01:44 INF - 	features	268MB
0:01:44 INF - 	mbtiles	514MB
-rw-r--r-- 1 runner docker 53M Mar  1 01:36 run.jar

ℹ️ This Branch Logs 4d32b2e

0:00:00 DEB - argument: config=null (path to config file)
0:00:00 DEB - argument: area=rhode island (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet'))
0:00:00 INF - Using in-memory stats

0:00:00 INF [overall] - Starting...
0:00:00 DEB - argument: bounds=Env[-74.07 : -17.84, 21.34 : 43.55] (bounds)
0:00:00 DEB - argument: threads=2 (num threads)
0:00:00 DEB - argument: loginterval=10 seconds (time between logs)
0:00:00 DEB - argument: minzoom=0 (minimum zoom level)
0:00:00 DEB - argument: maxzoom=14 (maximum zoom level (limit 14))
0:00:00 DEB - argument: defer_mbtiles_index_creation=false (skip adding index to mbtiles file)
0:00:00 DEB - argument: optimize_db=false (optimize mbtiles after writing)
0:00:00 DEB - argument: emit_tiles_in_order=true (emit tiles in index order)
0:00:00 DEB - argument: force=false (force overwriting output file)
0:00:00 DEB - argument: gzip_temp=false (gzip temporary feature storage (uses more CPU, but less disk space))
0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks)
0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks)
0:00:00 DEB - argument: nodemap_type=sortedtable (type of node location map: noop, sortedtable, or sparsearray)
0:00:00 DEB - argument: nodemap_storage=mmap (storage for location map: mmap or ram)
0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP)
0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP)
0:00:00 DEB - argument: download_chunk_size_mb=100 (Size of file chunks to download in parallel in megabytes)
0:00:00 DEB - argument: download_threads=1 (Number of parallel threads to use when downloading each file)
0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level)
0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level)
0:00:00 DEB - argument: osm_lazy_reads=false (Read OSM blocks from disk in worker threads)
0:00:00 DEB - argument: tmpdir=data/tmp (temp directory)
0:00:00 DEB - argument: only_download=false (download source data then exit)
0:00:00 DEB - argument: download=false (download sources)
0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit)
0:00:00 DEB - argument: fetch_wikidata=false (fetch wikidata translations then continue)
0:00:00 DEB - argument: use_wikidata=true (use wikidata translations)
0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file)
0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path)
0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path)
0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path)
0:00:00 DEB - argument: osm_path=data/sources/rhode_island.osm.pbf (osm OSM input file path)
0:00:00 DEB - argument: mbtiles=data/out.mbtiles (mbtiles output file)
0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names)
0:00:00 DEB - argument: languages=am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh (languages to use)
0:00:00 DEB - argument: only_layers= (Include only certain layers)
0:00:00 DEB - argument: exclude_layers= (Exclude certain layers)
0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries)
0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13)
0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13)
0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways)
0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield))
0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways)
0:00:00 DEB - argument: mbtiles_name=OpenMapTiles ('name' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a> ('attribution' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_version=3.13.0 ('version' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_type=baselayer ('type' attribute for mbtiles metadata)
0:00:00 DEB - argument: help=false (show arguments then exit)
0:00:00 INF - Building BasemapProfile profile into data/out.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   mbtiles: Encode each tile and write to data/out.mbtiles
0:00:00 INF - error loading /home/runner/work/planetiler/planetiler/data/sources/wikidata_names.json: java.nio.file.NoSuchFileException: data/sources/wikidata_names.json
0:00:00 INF - Using merge sort feature map, chunk size=1000mb workers=2
0:00:02 INF - dataFileCache open start

0:00:02 INF [lake_centerlines] - Starting...
0:00:02 INF [lake_centerlines] -  read: [  14k 100%  36k/s ] write: [    0    0/s ] 0    
    cpus: 1.7 gc:  0% mem: 162M/4.2G postGC: 46M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:02 INF [lake_centerlines] -  read: [  14k 100%    0/s ] write: [    0    0/s ] 0    
    cpus: 1.5 gc:  0% mem: 162M/4.2G postGC: 46M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:02 INF [lake_centerlines] - Finished in 0.5s cpu:0.8s avg:1.7
0:00:02 INF [lake_centerlines] -   read     1x(59% 0.3s)
0:00:02 INF [lake_centerlines] -   process  2x(16% 0.1s)
0:00:02 INF [lake_centerlines] -   write    1x(0% 0s)

0:00:02 INF [water_polygons] - Starting...
0:00:12 INF [water_polygons] -  read: [ 2.2k  16%  227/s ] write: [  60k 5.9k/s ] 4.2M 
    cpus: 1.9 gc: 11% mem: 1.2G/4.2G postGC: 1.1G
    read(58%) ->  (277/1k) -> process(37% 21%) -> (757/53k) -> write( 0%)
0:00:22 INF [water_polygons] -  read: [ 9.5k  66%  725/s ] write: [ 1.8M 178k/s ] 80M  
    cpus: 1.6 gc:  9% mem: 2.8G/4.2G postGC: 1.6G
    read(69%) ->   (1k/1k) -> process(19% 21%) -> (482/53k) -> write( 3%)
0:00:24 INF [water_polygons] -  read: [  14k 100% 3.6k/s ] write: [ 4.3M 1.8M/s ] 186M 
    cpus: 2 gc: 11% mem: 2.1G/4.2G postGC: 1.6G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:24 INF [water_polygons] -  read: [  14k 100%    0/s ] write: [ 4.3M    0/s ] 186M 
    cpus: 2.1 gc:  0% mem: 2.1G/4.2G postGC: 1.6G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:24 INF [water_polygons] - Finished in 21s cpu:38s gc:2s avg:1.8
0:00:24 INF [water_polygons] -   read     1x(60% 13s wait:2s)
0:00:24 INF [water_polygons] -   process  2x(26% 6s wait:10s)
0:00:24 INF [water_polygons] -   write    1x(3% 0.7s wait:20s)
0:00:24 INF [natural_earth] - unzipping /home/runner/work/planetiler/planetiler/data/sources/natural_earth_vector.sqlite.zip to data/tmp/natearth.sqlite

0:00:31 INF [natural_earth] - Starting...
0:00:39 INF [natural_earth] -  read: [ 349k 100%  46k/s ] write: [  181   24/s ] 186M 
    cpus: 1.6 gc:  2% mem: 3.3G/4.2G postGC: 1.6G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:39 INF [natural_earth] -  read: [ 349k 100%    0/s ] write: [  181    0/s ] 186M 
    cpus: 3.6 gc:  0% mem: 3.3G/4.2G postGC: 1.6G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:39 INF [natural_earth] - Finished in 8s cpu:13s avg:1.6
0:00:39 INF [natural_earth] -   read     1x(86% 7s sys:1s)
0:00:39 INF [natural_earth] -   process  2x(19% 2s wait:7s)
0:00:39 INF [natural_earth] -   write    1x(0% 0s wait:7s)

0:00:40 INF [osm_pass1] - Starting...
0:00:43 INF [osm_pass1] -  nodes: [ 4.5M 1.4M/s ] 353M  ways: [ 326k 107k/s ] rels: [ 7.3k 2.4k/s ] blocks: [  614  201/s ]
    cpus: 1.8 gc:  3% mem: 757M/4.2G postGC: 781M hppc: 882k
    read( -%) ->     (0/4) -> parse( -%) ->     (0/4) -> process( -%)
0:00:43 DEB [osm_pass1] - processed blocks:614 nodes:4,572,227 ways:326,909 relations:7,387
0:00:43 INF [osm_pass1] - Finished in 3s cpu:6s avg:1.8
0:00:43 INF [osm_pass1] -   read     1x(3% 0.1s wait:3s)
0:00:43 INF [osm_pass1] -   parse    1x(67% 2s)
0:00:43 INF [osm_pass1] -   process  1x(34% 1s wait:1s)

0:00:43 INF [osm_pass2] - Starting...
0:00:46 DEB [osm_pass2:process] - Sorting long long multimap...
0:00:46 DEB [osm_pass2:process] - Sorted long long multimap time:0.013250952s
0:00:46 WAR [osm_pass2:process] - No GB polygon for inferring route network types
0:00:53 INF [osm_pass2] -  nodes: [ 4.5M 100% 457k/s ] 353M  ways: [ 176k  54%  17k/s ] rels: [    0   0%    0/s ] features: [ 4.9M  63k/s ] 230M  blocks: [  594  97%   59/s ]
    cpus: 2 gc:  1% mem: 1.1G/4.2G postGC: 804M hppc: 1.6M
    read( -%) ->  (18/103) -> process(65% 65%) -> (672/53k) -> write( 2%)
0:01:03 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 353M  ways: [ 326k 100%  15k/s ] rels: [ 3.3k  45%  334/s ] features: [ 5.2M  29k/s ] 258M  blocks: [  613 100%    1/s ]
    cpus: 2 gc:  1% mem: 1.6G/4.2G postGC: 821M hppc:  29M
    read( -%) ->   (0/103) -> process(83% 78%) -> (366/53k) -> write( 1%)
0:01:08 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 353M  ways: [ 326k 100%    0/s ] rels: [ 7.3k 100%  732/s ] features: [ 5.2M 3.5k/s ] 268M  blocks: [  614 100%   <1/s ]
    cpus: 2 gc:  1% mem: 1.6G/4.2G postGC: 812M hppc:  29M
    read( -%) ->   (0/103) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:01:08 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 353M  ways: [ 326k 100%    0/s ] rels: [ 7.3k 100%    0/s ] features: [ 5.2M    0/s ] 268M  blocks: [  614 100%    0/s ]
    cpus: 0 gc:  0% mem: 1.6G/4.2G postGC: 812M hppc:  29M
    read( -%) ->   (0/103) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:01:08 DEB [osm_pass2] - processed blocks:614 nodes:4,572,227 ways:326,909 relations:7,387
0:01:08 INF [osm_pass2] - Finished in 26s cpu:51s avg:2
0:01:08 INF [osm_pass2] -   read     1x(0% 0s wait:3s done:23s)
0:01:08 INF [osm_pass2] -   process  2x(75% 19s)
0:01:08 INF [osm_pass2] -   write    1x(1% 0.3s wait:25s)

0:01:08 INF [boundaries] - Starting...
0:01:08 INF [boundaries] - Creating polygons for 1 boundaries
0:01:08 WAR [boundaries] - Unable to form closed polygon for OSM relation 148838 (likely missing edges)
0:01:08 INF [boundaries] - Finished creating 0 country polygons
0:01:08 INF [boundaries] - Finished in 0s cpu:0.1s avg:1.7
0:01:08 INF - Deleting node.db to make room for mbtiles

0:01:08 INF [sort] - Starting...
0:01:12 INF [sort] -  chunks: [   1 /   1 100% ] 268M 
    cpus: 1.2 gc:  0% mem: 2.1G/4.2G postGC: 812M
    ->     (0/4) -> worker(-0%  -%)
0:01:12 INF [sort] -  chunks: [   1 /   1 100% ] 268M 
    cpus: 0 gc:  0% mem: 2.1G/4.2G postGC: 812M
    ->     (0/4) -> worker( -%  -%)
0:01:12 INF [sort] - Finished in 4s cpu:5s avg:1.2
0:01:12 INF [sort] -   worker  2x(43% 2s done:2s)
0:01:12 INF - read:1s write:1s sort:0s

0:01:12 INF [mbtiles] - Starting...
0:01:13 DEB [mbtiles:write] - Execute mbtiles: create table metadata (name text, value text);
0:01:13 DEB [mbtiles:write] - Execute mbtiles: create unique index name on metadata (name);
0:01:13 DEB [mbtiles:write] - Execute mbtiles: create table tiles (zoom_level integer, tile_column integer, tile_row, tile_data blob);
0:01:13 DEB [mbtiles:write] - Execute mbtiles: create unique index tile_index on tiles (zoom_level, tile_column, tile_row)
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: name=OpenMapTiles
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: format=pbf
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: version=3.13.0
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: type=baselayer
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: bounds=-74.07,21.34,-17.84,43.55
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: center=-45.955,32.445,3
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: minzoom=0
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: maxzoom=14
0:01:13 DEB [mbtiles:write] - Set mbtiles metadata: json={"vector_layers":[{"id":"aerodrome_label","fields":{"name_int":"String","iata":"String","ele_ft":"Number","name_de":"String","name":"String","icao":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":10,"maxzoom":14},{"id":"aeroway","fields":{"ref":"String","class":"String"},"minzoom":10,"maxzoom":14},{"id":"boundary","fields":{"disputed":"Number","admin_level":"Number","maritime":"Number","disputed_name":"String"},"minzoom":0,"maxzoom":14},{"id":"building","fields":{"colour":"String","render_height":"Number","render_min_height":"Number"},"minzoom":13,"maxzoom":14},{"id":"housenumber","fields":{"housenumber":"String"},"minzoom":14,"maxzoom":14},{"id":"landcover","fields":{"subclass":"String","class":"String","_numpoints":"Number"},"minzoom":8,"maxzoom":14},{"id":"landuse","fields":{"class":"String"},"minzoom":4,"maxzoom":14},{"id":"mountain_peak","fields":{"name_int":"String","customary_ft":"Number","ele_ft":"Number","name_de":"String","name":"String","rank":"Number","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":7,"maxzoom":14},{"id":"park","fields":{"name_int":"String","name_de":"String","name":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"place","fields":{"name:fy":"String","name_int":"String","capital":"Number","name:uk":"String","name:pl":"String","name:nl":"String","name:be":"String","name:ru":"String","name:ko":"String","name_de":"String","name":"String","rank":"Number","name:en":"String","name:eo":"String","class":"String","name:hu":"String","name:ta":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":2,"maxzoom":14},{"id":"poi","fields":{"name_int":"String","level":"Number","name:nonlatin":"String","layer":"Number","name_de":"String","name":"String","subclass":"String","indoor":"Number","name:en":"String","class":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":12,"maxzoom":14},{"id":"transportation","fields":{"access":"String","brunnel":"String","expressway":"Number","surface":"String","bicycle":"String","level":"Number","ramp":"Number","mtb_scale":"String","toll":"Number","oneway":"Number","layer":"Number","network":"String","horse":"String","service":"String","subclass":"String","class":"String","foot":"String"},"minzoom":4,"maxzoom":14},{"id":"transportation_name","fields":{"name_int":"String","name:nonlatin":"String","route_4":"String","route_3":"String","route_2":"String","route_1":"String","layer":"Number","network":"String","ref":"String","name_de":"String","name":"String","subclass":"String","ref_length":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"water","fields":{"intermittent":"Number","class":"String"},"minzoom":0,"maxzoom":14},{"id":"water_name","fields":{"name_int":"String","name:nonlatin":"String","name_de":"String","name":"String","intermittent":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":9,"maxzoom":14},{"id":"waterway","fields":{"name_int":"String","brunnel":"String","name_de":"String","_relid":"Number","intermittent":"Number","name":"String","class":"String","name:latin":"String","name_en":"String"},"minzoom":4,"maxzoom":14}]}
0:01:13 INF [mbtiles:write] - Starting z0
0:01:13 INF [mbtiles:write] - Finished z0 in 0s cpu:0s avg:0, now starting z1
0:01:13 INF [mbtiles:write] - Finished z1 in 0s cpu:0s avg:0, now starting z2
0:01:13 INF [mbtiles:write] - Finished z2 in 0s cpu:0s avg:0, now starting z3
0:01:13 INF [mbtiles:write] - Finished z3 in 0s cpu:0s avg:0, now starting z4
0:01:13 INF [mbtiles:write] - Finished z4 in 0s cpu:0s avg:0, now starting z5
0:01:13 INF [mbtiles:write] - Finished z5 in 0s cpu:0s avg:0, now starting z6
0:01:13 INF [mbtiles:write] - Finished z6 in 0s cpu:0s avg:0, now starting z7
0:01:13 INF [mbtiles:write] - Finished z7 in 0.1s cpu:0.1s avg:1.9, now starting z8
0:01:14 INF [mbtiles:write] - Finished z8 in 1s cpu:2s avg:2, now starting z9
0:01:15 INF [mbtiles:write] - Finished z9 in 0.7s cpu:1s avg:1.9, now starting z10
0:01:15 INF [mbtiles:write] - Finished z10 in 0.3s cpu:0.7s avg:2, now starting z11
0:01:16 INF [mbtiles:write] - Finished z11 in 0.4s cpu:0.8s avg:2, now starting z12
0:01:18 INF [mbtiles:write] - Finished z12 in 2s cpu:4s avg:2, now starting z13
0:01:23 INF [mbtiles] -  features: [ 727k  14%  72k/s ] 268M  tiles: [ 291k  29k/s ] 46M  
    cpus: 2 gc:  4% mem: 3G/4.2G postGC: 833M
    read( 3%) -> (214/217) -> encode(58% 56%) -> (215/216) -> write( 7%)
    last tile: 13/2470/3047 (z13 4%) https://www.openstreetmap.org/#map=13/41.80408/-71.45508
0:01:29 INF [mbtiles:write] - Finished z13 in 11s cpu:22s avg:2, now starting z14
0:01:33 INF [mbtiles] -  features: [ 3.4M  65% 271k/s ] 268M  tiles: [ 2.1M 183k/s ] 280M 
    cpus: 1.8 gc:  2% mem: 1.7G/4.2G postGC: 826M
    read( 8%) ->   (0/217) -> encode(48% 47%) -> (214/216) -> write(41%)
    last tile: 14/5740/6649 (z14 35%) https://www.openstreetmap.org/#map=14/32.08257/-53.87695
0:01:37 INF [mbtiles:write] - Finished z14 in 8s cpu:12s avg:1.5
0:01:37 INF [mbtiles] -  features: [ 5.2M 100% 435k/s ] 268M  tiles: [ 4.1M 472k/s ] 514M 
    cpus: 1.4 gc:  0% mem: 2.6G/4.2G postGC: 826M
    read( -%) ->   (0/217) -> encode( -%  -%) ->   (0/216) -> write( -%)
    last tile: 14/7380/5985 (z14 100%) https://www.openstreetmap.org/#map=14/43.56447/-17.84180
0:01:37 DEB [mbtiles] - Tile stats:
0:01:37 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:37 DEB [mbtiles] - z1 avg:4k max:4k
0:01:37 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:37 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:37 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:37 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:37 DEB [mbtiles] - z6 avg:1.4k max:24k
0:01:37 DEB [mbtiles] - z7 avg:896 max:33k
0:01:37 DEB [mbtiles] - z8 avg:365 max:48k
0:01:37 DEB [mbtiles] - z9 avg:295 max:279k
0:01:37 DEB [mbtiles] - z10 avg:164 max:232k
0:01:37 DEB [mbtiles] - z11 avg:107 max:131k
0:01:37 DEB [mbtiles] - z12 avg:85 max:118k
0:01:37 DEB [mbtiles] - z13 avg:72 max:109k
0:01:37 DEB [mbtiles] - z14 avg:68 max:257k
0:01:37 DEB [mbtiles] - all avg:70 max:0
0:01:37 DEB [mbtiles] -  # features: 5,275,803
0:01:37 DEB [mbtiles] -     # tiles: 4,115,465
0:01:37 INF [mbtiles] - Finished in 25s cpu:45s avg:1.8
0:01:37 INF [mbtiles] -   read    1x(8% 2s wait:20s)
0:01:37 INF [mbtiles] -   encode  2x(44% 11s wait:6s)
0:01:37 INF [mbtiles] -   write   1x(37% 9s wait:13s)
0:01:37 INF - Finished in 1m37s cpu:2m47s gc:4s avg:1.7
0:01:37 INF - FINISHED!

0:01:37 INF - ----------------------------------------
0:01:37 INF - 	overall          1m37s cpu:2m47s gc:4s avg:1.7
0:01:37 INF - 	lake_centerlines 0.5s cpu:0.8s avg:1.7
0:01:37 INF - 	water_polygons   21s cpu:38s gc:2s avg:1.8
0:01:37 INF - 	  read     1x(60% 13s wait:2s)
0:01:37 INF - 	  process  2x(26% 6s wait:10s)
0:01:37 INF - 	  write    1x(3% 0.7s wait:20s)
0:01:37 INF - 	natural_earth    8s cpu:13s avg:1.6
0:01:37 INF - 	  read     1x(86% 7s sys:1s)
0:01:37 INF - 	  process  2x(19% 2s wait:7s)
0:01:37 INF - 	  write    1x(0% 0s wait:7s)
0:01:37 INF - 	osm_pass1        3s cpu:6s avg:1.8
0:01:37 INF - 	  read     1x(3% 0.1s wait:3s)
0:01:37 INF - 	  parse    1x(67% 2s)
0:01:37 INF - 	  process  1x(34% 1s wait:1s)
0:01:37 INF - 	osm_pass2        26s cpu:51s avg:2
0:01:37 INF - 	  read     1x(0% 0s wait:3s done:23s)
0:01:37 INF - 	  process  2x(75% 19s)
0:01:37 INF - 	  write    1x(1% 0.3s wait:25s)
0:01:37 INF - 	boundaries       0s cpu:0.1s avg:1.7
0:01:37 INF - 	sort             4s cpu:5s avg:1.2
0:01:37 INF - 	  worker  2x(43% 2s done:2s)
0:01:37 INF - 	mbtiles          25s cpu:45s avg:1.8
0:01:37 INF - 	  read    1x(8% 2s wait:20s)
0:01:37 INF - 	  encode  2x(44% 11s wait:6s)
0:01:37 INF - 	  write   1x(37% 9s wait:13s)
0:01:37 INF - ----------------------------------------
0:01:37 INF - 	features	268MB
0:01:37 INF - 	mbtiles	514MB
-rw-r--r-- 1 runner docker 53M Mar  1 01:34 run.jar

This reverts commit 31efaac.

msbarry · 2022-03-01T01:47:24Z

Ran pass1/pass2 on the planet on a 16-core digitalocean droplet wtih 128GB ram. Looks like it shaves about 12 minutes off pass1, and pass2 is almost the same but pass2 should saturate more (40+) cores better although I haven't tested it.

Main:

INF [osm_pass1] - Finished in 17m49s cpu:1h19m15s gc:47s avg:4.4
INF [osm_pass1] -   pbf      1x(67% cpu:11m52s sys:28s wait:5m6s)
INF [osm_pass1] -   process  1x(93% cpu:16m39s sys:4s wait:16s)
...
INF [osm_pass2] - Finished in 1h6m38s cpu:16h17m42s gc:1m49s avg:14.7
INF [osm_pass2] -   pbf      1x(11% cpu:7m28s sys:40s wait:57m57s done:4s)
INF [osm_pass2] -   process 15x(85% cpu:56m36s sys:5s block:2m13s wait:3m9s)
INF [osm_pass2] -   write    1x(29% cpu:19m36s sys:3m33s wait:45m44s)

This branch:

INF [osm_pass1] - Finished in 6m12s cpu:59m4s gc:34s avg:9.5
INF [osm_pass1] -   read     1x(12% 46s sys:29s wait:5m5s done:2s)
INF [osm_pass1] -   parse   14x(51% 3m10s wait:2m39s)
INF [osm_pass1] -   process  1x(86% 5m20s wait:17s)
...
INF [osm_pass2] - Finished in 1h7m39s cpu:16h57m24s gc:3m22s avg:15
INF [osm_pass2] -   read     1x(1% 51s sys:34s wait:58m54s done:7m19s)
INF [osm_pass2] -   process 15x(90% 1h1m9s sys:3s block:40s wait:6s done:17s)
INF [osm_pass2] -   write    1x(28% 19m1s sys:3m32s wait:47m13s)

msbarry added 2 commits February 17, 2022 06:03

Custom PBF parsing

2789491

delete files

0ad5a7b

msbarry added 25 commits February 23, 2022 05:28

log element counts

e267f5f

fix counts

d5b3d42

tweak logs and min threads

3454a7d

Merge branch 'main' into custom-pbf-parsing

02d78ae

fix

25272a7

logging tweaks

3d10716

Merge branch 'main' into custom-pbf-parsing

f6725aa

hold off on removing graphhopper dependency

41dbdca

undo notice.md change

aa62729

only log gc when there is gc

919487c

ensure enough bytes are read

b37c40a

tests for pbf parser/decoder

d3d646c

tweak OsmInputFile organization

1964de9

cleanup and parse more header

a35fc9e

check expecations about OSM input file

cc7fbd7

more compact summary

31efaac

Revert "more compact summary"

b5aab15

This reverts commit 31efaac.

skip summary on short steps

344d715

fix iterable once

0587ead

fix

fd9d6f0

reduce pending batches

b863ab7

tweak PbfDecoder

c84751d

reduce GC pressure with WeightedHandoffQueue

5f03b1d

turn off color logging for blocks

310460e

distributor

45a7e58

msbarry merged commit 2f05f94 into main Mar 1, 2022

msbarry deleted the custom-pbf-parsing branch March 1, 2022 01:52

msbarry mentioned this pull request Mar 1, 2022

Remove graphhopper dependency #101

Merged

wipfli mentioned this pull request Mar 4, 2022

Add intellij code formatter workflow wipfli/planetiler#1

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Custom PBF parsing #82

Custom PBF parsing #82

msbarry commented Feb 17, 2022 •

edited

github-actions bot commented Feb 17, 2022 •

edited

msbarry commented Mar 1, 2022 •

edited

Custom PBF parsing #82

Custom PBF parsing #82

Conversation

msbarry commented Feb 17, 2022 • edited

github-actions bot commented Feb 17, 2022 • edited

msbarry commented Mar 1, 2022 • edited

msbarry commented Feb 17, 2022 •

edited

github-actions bot commented Feb 17, 2022 •

edited

msbarry commented Mar 1, 2022 •

edited