Offload multipolygon storage to disk (reduce memory usage by 10-15gb) #141

msbarry · 2022-03-22T00:21:46Z

The next biggest object held in memory after node locations is multipolygon way geometries. This was storing node IDs in an HPPC LongLongMap which makes a copy every time it expands. The list was getting up to 5-7GB, which requires an extra 5-7GB each time it expands. This change moves that to a more memory-efficient AppendStore that doesn't require an extra 5-7GB memory to expand, and can also be offloaded to disk save another 5-7GB.

I tested this out on the planet twice:

-Xmx32g on a 16 cpu/64GB digitalocean droplet: finished in 2h19m (osm_pass2 took 1h36m)
-Xmx20g on a 16 cpu/32GB digitalocean droplet: finished in 2h58m (osm_pass2 took 2h5m)

Checklist:

expose settings so you can switch the append store implementation
estimate memory usage
report memory usage in logs

github-actions · 2022-03-22T00:28:42Z

Base 313a695 This Branch f80e6dd

0:01:37 DEB [mbtiles] - Tile stats:
0:01:37 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:37 DEB [mbtiles] - z1 avg:4k max:4k
0:01:37 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:37 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:37 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:37 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:37 DEB [mbtiles] - z6 avg:1.4k max:24k
0:01:37 DEB [mbtiles] - z7 avg:899 max:33k
0:01:37 DEB [mbtiles] - z8 avg:366 max:48k
0:01:37 DEB [mbtiles] - z9 avg:296 max:278k
0:01:37 DEB [mbtiles] - z10 avg:164 max:232k
0:01:37 DEB [mbtiles] - z11 avg:107 max:132k
0:01:37 DEB [mbtiles] - z12 avg:86 max:118k
0:01:37 DEB [mbtiles] - z13 avg:72 max:109k
0:01:37 DEB [mbtiles] - z14 avg:68 max:256k
0:01:37 DEB [mbtiles] - all avg:70 max:0
0:01:37 DEB [mbtiles] -  # features: 5,277,787
0:01:37 DEB [mbtiles] -     # tiles: 4,115,423
0:01:37 INF [mbtiles] - Finished in 23s cpu:43s avg:1.8
0:01:37 INF [mbtiles] -   read    1x(7% 2s wait:19s)
0:01:37 INF [mbtiles] -   encode  2x(45% 10s wait:6s)
0:01:37 INF [mbtiles] -   write   1x(36% 8s wait:12s)
0:01:37 INF - Finished in 1m37s cpu:2m52s gc:3s avg:1.8
0:01:37 INF - FINISHED!
0:01:37 INF - 
0:01:37 INF - ----------------------------------------
0:01:37 INF - 	overall          1m37s cpu:2m52s gc:3s avg:1.8
0:01:37 INF - 	lake_centerlines 1s cpu:3s avg:1.9
0:01:37 INF - 	  read     1x(80% 1s)
0:01:37 INF - 	  process  2x(13% 0.2s wait:1s)
0:01:37 INF - 	  write    1x(0% 0s wait:1s)
0:01:37 INF - 	water_polygons   23s cpu:40s gc:2s avg:1.8
0:01:37 INF - 	  read     1x(61% 14s wait:2s)
0:01:37 INF - 	  process  2x(27% 6s wait:11s)
0:01:37 INF - 	  write    1x(4% 0.9s wait:22s)
0:01:37 INF - 	natural_earth    8s cpu:13s avg:1.6
0:01:37 INF - 	  read     1x(89% 7s sys:1s)
0:01:37 INF - 	  process  2x(20% 2s wait:7s)
0:01:37 INF - 	  write    1x(0% 0s wait:8s)
0:01:37 INF - 	osm_pass1        3s cpu:7s avg:1.9
0:01:37 INF - 	  read     1x(3% 0.1s wait:3s)
0:01:37 INF - 	  parse    1x(62% 2s wait:1s)
0:01:37 INF - 	  process  1x(51% 2s wait:1s)
0:01:37 INF - 	osm_pass2        27s cpu:54s avg:2
0:01:37 INF - 	  read     1x(0% 0s wait:3s done:24s)
0:01:37 INF - 	  process  2x(74% 20s)
0:01:37 INF - 	  write    1x(1% 0.4s wait:27s)
0:01:37 INF - 	boundaries       0s cpu:0.1s avg:1.5
0:01:37 INF - 	sort             3s cpu:4s avg:1.2
0:01:37 INF - 	  worker  2x(44% 1s done:2s)
0:01:37 INF - 	mbtiles          23s cpu:43s avg:1.8
0:01:37 INF - 	  read    1x(7% 2s wait:19s)
0:01:37 INF - 	  encode  2x(45% 10s wait:6s)
0:01:37 INF - 	  write   1x(36% 8s wait:12s)
0:01:37 INF - ----------------------------------------
0:01:37 INF - 	features	268MB
0:01:37 INF - 	mbtiles	514MB
-rw-r--r-- 1 runner docker 55M Mar 23 00:29 run.jar

0:01:42 DEB [mbtiles] - Tile stats:
0:01:42 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:42 DEB [mbtiles] - z1 avg:4k max:4k
0:01:42 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:42 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:42 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:42 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:42 DEB [mbtiles] - z6 avg:1.4k max:24k
0:01:42 DEB [mbtiles] - z7 avg:899 max:33k
0:01:42 DEB [mbtiles] - z8 avg:366 max:48k
0:01:42 DEB [mbtiles] - z9 avg:296 max:278k
0:01:42 DEB [mbtiles] - z10 avg:164 max:232k
0:01:42 DEB [mbtiles] - z11 avg:107 max:132k
0:01:42 DEB [mbtiles] - z12 avg:86 max:118k
0:01:42 DEB [mbtiles] - z13 avg:72 max:109k
0:01:42 DEB [mbtiles] - z14 avg:68 max:256k
0:01:42 DEB [mbtiles] - all avg:70 max:0
0:01:42 DEB [mbtiles] -  # features: 5,277,787
0:01:42 DEB [mbtiles] -     # tiles: 4,115,423
0:01:42 INF [mbtiles] - Finished in 23s cpu:42s avg:1.8
0:01:42 INF [mbtiles] -   read    1x(8% 2s wait:19s)
0:01:42 INF [mbtiles] -   encode  2x(44% 10s wait:6s)
0:01:42 INF [mbtiles] -   write   1x(37% 9s sys:1s wait:12s)
0:01:42 INF - Finished in 1m42s cpu:2m50s gc:3s avg:1.7
0:01:42 INF - FINISHED!
0:01:42 INF - 
0:01:42 INF - ----------------------------------------
0:01:42 INF - 	overall          1m42s cpu:2m50s gc:3s avg:1.7
0:01:42 INF - 	lake_centerlines 1s cpu:3s avg:1.8
0:01:42 INF - 	  read     1x(78% 1s)
0:01:42 INF - 	  process  2x(14% 0.2s wait:1s)
0:01:42 INF - 	  write    1x(0% 0s wait:1s)
0:01:42 INF - 	water_polygons   24s cpu:41s gc:2s avg:1.7
0:01:42 INF - 	  read     1x(60% 14s wait:2s)
0:01:42 INF - 	  process  2x(26% 6s wait:11s)
0:01:42 INF - 	  write    1x(4% 0.8s wait:23s)
0:01:42 INF - 	natural_earth    9s cpu:14s avg:1.5
0:01:42 INF - 	  read     1x(78% 7s sys:1s done:2s)
0:01:42 INF - 	  process  2x(17% 2s wait:7s done:2s)
0:01:42 INF - 	  write    1x(0% 0s wait:8s done:2s)
0:01:42 INF - 	osm_pass1        3s cpu:5s avg:1.8
0:01:42 INF - 	  read     1x(3% 0.1s wait:3s)
0:01:42 INF - 	  parse    1x(63% 2s)
0:01:42 INF - 	  process  1x(49% 1s wait:1s)
0:01:42 INF - 	osm_pass2        26s cpu:52s avg:2
0:01:42 INF - 	  read     1x(0% 0s wait:13s done:13s)
0:01:42 INF - 	  process  2x(76% 20s)
0:01:42 INF - 	  write    1x(1% 0.4s wait:26s)
0:01:42 INF - 	boundaries       0s cpu:0.1s avg:1.5
0:01:42 INF - 	sort             4s cpu:4s avg:1.1
0:01:42 INF - 	  worker  2x(38% 2s done:2s)
0:01:42 INF - 	mbtiles          23s cpu:42s avg:1.8
0:01:42 INF - 	  read    1x(8% 2s wait:19s)
0:01:42 INF - 	  encode  2x(44% 10s wait:6s)
0:01:42 INF - 	  write   1x(37% 9s sys:1s wait:12s)
0:01:42 INF - ----------------------------------------
0:01:42 INF - 	features	268MB
0:01:42 INF - 	mbtiles	514MB
-rw-r--r-- 1 runner docker 55M Mar 23 00:27 run.jar

https://github.com/onthegomap/planetiler/actions/runs/2025624986

ℹ️ Base Logs 313a695

0:00:00 DEB - argument: config=null (path to config file)
0:00:00 DEB - argument: area=rhode island (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet'))
0:00:00 INF - Using in-memory stats
0:00:00 INF [overall] - 
0:00:00 INF [overall] - Starting...
0:00:00 DEB - argument: bounds=Env[-74.07 : -17.84, 21.34 : 43.55] (bounds)
0:00:00 DEB - argument: threads=2 (num threads)
0:00:00 DEB - argument: loginterval=10 seconds (time between logs)
0:00:00 DEB - argument: minzoom=0 (minimum zoom level)
0:00:00 DEB - argument: maxzoom=14 (maximum zoom level (limit 14))
0:00:00 DEB - argument: defer_mbtiles_index_creation=false (skip adding index to mbtiles file)
0:00:00 DEB - argument: optimize_db=false (optimize mbtiles after writing)
0:00:00 DEB - argument: emit_tiles_in_order=true (emit tiles in index order)
0:00:00 DEB - argument: force=false (overwriting output file and ignore disk/RAM warnings)
0:00:00 DEB - argument: gzip_temp=false (gzip temporary feature storage (uses more CPU, but less disk space))
0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks)
0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks)
0:00:00 DEB - argument: nodemap_type=sparsearray (type of node location map, one of [noop, sortedtable, sparsearray, array])
0:00:00 DEB - argument: nodemap_storage=mmap (storage for location map, one of [ram, mmap, direct])
0:00:00 DEB - argument: nodemap_madvise=false (use linux madvise(random) to improve memory-mapped read performance)
0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP)
0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP)
0:00:00 DEB - argument: http_retries=1 (Retries to use when downloading files over HTTP)
0:00:00 DEB - argument: download_chunk_size_mb=100 (Size of file chunks to download in parallel in megabytes)
0:00:00 DEB - argument: download_threads=1 (Number of parallel threads to use when downloading each file)
0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level)
0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level)
0:00:00 DEB - argument: osm_lazy_reads=false (Read OSM blocks from disk in worker threads)
0:00:00 DEB - argument: tmpdir=data/tmp (temp directory)
0:00:00 DEB - argument: only_download=false (download source data then exit)
0:00:00 DEB - argument: download=false (download sources)
0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit)
0:00:00 DEB - argument: fetch_wikidata=false (fetch wikidata translations then continue)
0:00:00 DEB - argument: use_wikidata=true (use wikidata translations)
0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file)
0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path)
0:00:00 DEB - argument: free_lake_centerlines_after_read=false (delete lake_centerlines input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path)
0:00:00 DEB - argument: free_water_polygons_after_read=false (delete water_polygons input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path)
0:00:00 DEB - argument: free_natural_earth_after_read=false (delete natural_earth input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: osm_path=data/sources/rhode_island.osm.pbf (osm OSM input file path)
0:00:00 DEB - argument: free_osm_after_read=false (delete osm input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: mbtiles=data/out.mbtiles (mbtiles output file)
0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names)
0:00:00 DEB - argument: languages=am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh (languages to use)
0:00:00 DEB - argument: only_layers= (Include only certain layers)
0:00:00 DEB - argument: exclude_layers= (Exclude certain layers)
0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries)
0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13)
0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13)
0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways)
0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield))
0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways)
0:00:00 DEB - argument: mbtiles_name=OpenMapTiles ('name' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a> ('attribution' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_version=3.13.0 ('version' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_type=baselayer ('type' attribute for mbtiles metadata)
0:00:00 DEB - argument: help=false (show arguments then exit)
0:00:00 INF - Building BasemapProfile profile into data/out.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   mbtiles: Encode each tile and write to data/out.mbtiles
0:00:00 INF - error loading /home/runner/work/planetiler/planetiler/data/sources/wikidata_names.json: java.nio.file.NoSuchFileException: data/sources/wikidata_names.json
0:00:00 DEB - ✓ 187M storage on / (/dev/root) requested for read phase disk, 32G available
0:00:00 DEB -  - 43M used for temporary node location cache
0:00:00 DEB -  - 143M used for temporary feature storage
0:00:00 DEB - ✓ 215M storage on / (/dev/root) requested for write phase disk, 32G available
0:00:00 DEB -  - 143M used for temporary feature storage
0:00:00 DEB -  - 71M used for mbtiles output
0:00:00 DEB - ✓ 321M JVM heap requested for read phase, 4.2G available
0:00:00 DEB -  - 300M used for sparsearray node location in-memory index
0:00:00 DEB -  - 21M used for temporary profile storage
0:00:00 DEB - ✓ 43M storage on / (/dev/root) requested for read phase, 32G available
0:00:00 DEB -  - 43M used for sparsearray node location cache
0:00:00 INF - Using merge sort feature map, chunk size=1000mb workers=2
0:00:01 INF - dataFileCache open start
0:00:01 INF [lake_centerlines] - 
0:00:01 INF [lake_centerlines] - Starting...
0:00:03 INF [lake_centerlines] -  read: [  51k 100%  36k/s ] write: [    0    0/s ] 0    
    cpus: 1.9 gc:  4% heap: 139M/4.2G direct: 73k postGC: 75M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:03 INF [lake_centerlines] - Finished in 1s cpu:3s avg:1.9
0:00:03 INF [lake_centerlines] -   read     1x(80% 1s)
0:00:03 INF [lake_centerlines] -   process  2x(13% 0.2s wait:1s)
0:00:03 INF [lake_centerlines] -   write    1x(0% 0s wait:1s)
0:00:03 INF [water_polygons] - 
0:00:03 INF [water_polygons] - Starting...
0:00:13 INF [water_polygons] -  read: [ 2.2k  16%  224/s ] write: [  77k 7.6k/s ] 5M   
    cpus: 2 gc: 10% heap: 1.9G/4.2G direct: 54M postGC: 1.1G
    read(58%) ->  (273/1k) -> process(43% 17%) -> (701/53k) -> write( 0%)
0:00:23 INF [water_polygons] -  read: [ 9.2k  64%  698/s ] write: [ 446k  36k/s ] 22M  
    cpus: 1.6 gc:  6% heap: 2.6G/4.2G direct: 54M postGC: 1.4G
    read(79%) ->   (1k/1k) -> process(18% 15%) -> (875/53k) -> write( 1%)
0:00:25 INF [water_polygons] -  read: [  14k 100%   2k/s ] write: [ 4.3M 1.5M/s ] 186M 
    cpus: 1.9 gc: 10% heap: 3G/4.2G direct: 54M postGC: 1.5G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:25 INF [water_polygons] - Finished in 23s cpu:40s gc:2s avg:1.8
0:00:25 INF [water_polygons] -   read     1x(61% 14s wait:2s)
0:00:25 INF [water_polygons] -   process  2x(27% 6s wait:11s)
0:00:25 INF [water_polygons] -   write    1x(4% 0.9s wait:22s)
0:00:25 INF [natural_earth] - unzipping /home/runner/work/planetiler/planetiler/data/sources/natural_earth_vector.sqlite.zip to data/tmp/natearth.sqlite
0:00:31 INF [natural_earth] - 
0:00:31 INF [natural_earth] - Starting...
0:00:39 INF [natural_earth] -  read: [ 349k 100%  46k/s ] write: [  181   23/s ] 186M 
    cpus: 1.6 gc:  0% heap: 864M/4.2G direct: 54M postGC: 175M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:39 INF [natural_earth] - Finished in 8s cpu:13s avg:1.6
0:00:39 INF [natural_earth] -   read     1x(89% 7s sys:1s)
0:00:39 INF [natural_earth] -   process  2x(20% 2s wait:7s)
0:00:39 INF [natural_earth] -   write    1x(0% 0s wait:8s)
0:00:39 INF [osm_pass1] - 
0:00:39 INF [osm_pass1] - Starting...
0:00:41 INF [osm_pass1:process] - Finished nodes: 4,576,086 (1.8M/s) in 2s cpu:4s avg:1.8
0:00:42 INF [osm_pass1:process] - Finished ways: 326,826 (352k/s) in 0.9s cpu:2s avg:2
0:00:42 INF [osm_pass1:process] - Finished relations: 7,573 (155k/s) in 0s cpu:0.1s avg:1.8
0:00:42 INF [osm_pass1] -  nodes: [ 4.5M 1.3M/s ] 408M  ways: [ 326k  93k/s ] rels: [ 7.5k 2.1k/s ] blocks: [  615  176/s ]
    cpus: 1.9 gc:  0% heap: 1.1G/4.2G direct: 54M postGC: 829M hppc: 880k
    read( -%) ->     (0/4) -> parse( -%) ->     (0/4) -> process( -%)
0:00:42 DEB [osm_pass1] - Processed 615 blocks:
0:00:42 DEB [osm_pass1] -   nodes: 4,576,086 (1.8M/s) in 2s cpu:4s avg:1.8
0:00:42 DEB [osm_pass1] -   ways: 326,826 (352k/s) in 0.9s cpu:2s avg:2
0:00:42 DEB [osm_pass1] -   relations: 7,573 (155k/s) in 0s cpu:0.1s avg:1.8
0:00:42 INF [osm_pass1] - Finished in 3s cpu:7s avg:1.9
0:00:42 INF [osm_pass1] -   read     1x(3% 0.1s wait:3s)
0:00:42 INF [osm_pass1] -   parse    1x(62% 2s wait:1s)
0:00:42 INF [osm_pass1] -   process  1x(51% 2s wait:1s)
0:00:42 INF [osm_pass2] - 
0:00:42 INF [osm_pass2] - Starting...
0:00:46 DEB [osm_pass2:process] - Sorting long long multimap...
0:00:46 INF [osm_pass2:process] - Finished nodes: 4,576,086 (1.2M/s) in 4s cpu:7s avg:2
0:00:46 DEB [osm_pass2:process] - Sorted long long multimap 0s cpu:0s avg:2.2
0:00:46 WAR [osm_pass2:process] - No GB polygon for inferring route network types
0:00:52 INF [osm_pass2] -  nodes: [ 4.5M 100% 457k/s ] 408M  ways: [ 155k  48%  15k/s ] rels: [    0   0%    0/s ] features: [ 4.8M  57k/s ] 228M  blocks: [  591  96%   59/s ]
    cpus: 2 gc:  1% heap: 3.1G/4.2G direct: 54M postGC: 861M hppc: 1.6M
    read( -%) ->  (22/103) -> process(67% 67%) -> (1.4k/53k) -> write( 2%)
0:01:01 INF [osm_pass2:process] - Finished ways: 326,826 (22k/s) in 15s cpu:29s avg:2
0:01:02 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 408M  ways: [ 326k 100%  17k/s ] rels: [ 2.4k  33%  246/s ] features: [ 5.2M  35k/s ] 256M  blocks: [  614 100%    2/s ]
    cpus: 2 gc:  1% heap: 3.3G/4.2G direct: 54M postGC: 866M hppc:  29M
    read( -%) ->   (0/103) -> process(76% 77%) ->  (1k/53k) -> write( 1%)
0:01:07 INF [osm_pass2:process] - Finished relations: 7,573 (1.1k/s) in 7s cpu:14s avg:2
0:01:10 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 408M  ways: [ 326k 100%    0/s ] rels: [ 7.5k 100%  699/s ] features: [ 5.2M 3.4k/s ] 268M  blocks: [  615 100%   <1/s ]
    cpus: 2 gc:  1% heap: 2G/4.2G direct: 54M postGC: 868M hppc:  29M
    read( -%) ->   (0/103) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:01:10 DEB [osm_pass2] - Processed 615 blocks:
0:01:10 DEB [osm_pass2] -   nodes: 4,576,086 (1.2M/s) in 4s cpu:7s avg:2
0:01:10 DEB [osm_pass2] -   ways: 326,826 (22k/s) in 15s cpu:29s avg:2
0:01:10 DEB [osm_pass2] -   relations: 7,573 (1.1k/s) in 7s cpu:14s avg:2
0:01:10 INF [osm_pass2] - Finished in 27s cpu:54s avg:2
0:01:10 INF [osm_pass2] -   read     1x(0% 0s wait:3s done:24s)
0:01:10 INF [osm_pass2] -   process  2x(74% 20s)
0:01:10 INF [osm_pass2] -   write    1x(1% 0.4s wait:27s)
0:01:10 INF [boundaries] - 
0:01:10 INF [boundaries] - Starting...
0:01:10 INF [boundaries] - Creating polygons for 1 boundaries
0:01:10 WAR [boundaries] - Unable to form closed polygon for OSM relation 148838 (likely missing edges)
0:01:10 INF [boundaries] - Finished creating 0 country polygons
0:01:10 INF [boundaries] - Finished in 0s cpu:0.1s avg:1.5
0:01:10 INF - Deleting node.db to make room for output file
0:01:10 INF [sort] - 
0:01:10 INF [sort] - Starting...
0:01:13 INF [sort] -  chunks: [   1 /   1 100% ] 268M 
    cpus: 1.2 gc:  0% heap: 2.5G/4.2G direct: 54M postGC: 868M
    ->     (0/4) -> worker( -%  -%)
0:01:13 INF [sort] - Finished in 3s cpu:4s avg:1.2
0:01:13 INF [sort] -   worker  2x(44% 1s done:2s)
0:01:13 INF - read:1s write:1s sort:0s
0:01:13 INF [mbtiles] - 
0:01:13 INF [mbtiles] - Starting...
0:01:14 DEB [mbtiles:write] - Execute mbtiles: create table metadata (name text, value text);
0:01:14 DEB [mbtiles:write] - Execute mbtiles: create unique index name on metadata (name);
0:01:14 DEB [mbtiles:write] - Execute mbtiles: create table tiles (zoom_level integer, tile_column integer, tile_row, tile_data blob);
0:01:14 DEB [mbtiles:write] - Execute mbtiles: create unique index tile_index on tiles (zoom_level, tile_column, tile_row)
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: name=OpenMapTiles
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: format=pbf
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: version=3.13.0
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: type=baselayer
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: bounds=-74.07,21.34,-17.84,43.55
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: center=-45.955,32.445,3
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: minzoom=0
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: maxzoom=14
0:01:14 DEB [mbtiles:write] - Set mbtiles metadata: json={"vector_layers":[{"id":"aerodrome_label","fields":{"name_int":"String","iata":"String","ele_ft":"Number","name_de":"String","name":"String","icao":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":10,"maxzoom":14},{"id":"aeroway","fields":{"ref":"String","class":"String"},"minzoom":10,"maxzoom":14},{"id":"boundary","fields":{"disputed":"Number","admin_level":"Number","maritime":"Number","disputed_name":"String"},"minzoom":0,"maxzoom":14},{"id":"building","fields":{"colour":"String","render_height":"Number","render_min_height":"Number"},"minzoom":13,"maxzoom":14},{"id":"housenumber","fields":{"housenumber":"String"},"minzoom":14,"maxzoom":14},{"id":"landcover","fields":{"subclass":"String","class":"String","_numpoints":"Number"},"minzoom":8,"maxzoom":14},{"id":"landuse","fields":{"class":"String"},"minzoom":4,"maxzoom":14},{"id":"mountain_peak","fields":{"name_int":"String","customary_ft":"Number","ele_ft":"Number","name_de":"String","name":"String","rank":"Number","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":7,"maxzoom":14},{"id":"park","fields":{"name_int":"String","name_de":"String","name":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"place","fields":{"name:fy":"String","name_int":"String","capital":"Number","name:uk":"String","name:pl":"String","name:nl":"String","name:be":"String","name:ru":"String","name:ko":"String","name_de":"String","name":"String","rank":"Number","name:en":"String","name:eo":"String","class":"String","name:hu":"String","name:ta":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":2,"maxzoom":14},{"id":"poi","fields":{"name_int":"String","level":"Number","name:nonlatin":"String","layer":"Number","name_de":"String","name":"String","subclass":"String","indoor":"Number","name:en":"String","class":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":12,"maxzoom":14},{"id":"transportation","fields":{"access":"String","brunnel":"String","expressway":"Number","surface":"String","bicycle":"String","level":"Number","ramp":"Number","mtb_scale":"String","toll":"Number","oneway":"Number","layer":"Number","network":"String","horse":"String","service":"String","subclass":"String","class":"String","foot":"String"},"minzoom":4,"maxzoom":14},{"id":"transportation_name","fields":{"name_int":"String","name:nonlatin":"String","route_4":"String","route_3":"String","route_2":"String","route_1":"String","layer":"Number","network":"String","ref":"String","name_de":"String","name":"String","subclass":"String","ref_length":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"water","fields":{"intermittent":"Number","class":"String"},"minzoom":0,"maxzoom":14},{"id":"water_name","fields":{"name_int":"String","name:nonlatin":"String","name_de":"String","name":"String","intermittent":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":9,"maxzoom":14},{"id":"waterway","fields":{"name_int":"String","brunnel":"String","name_de":"String","_relid":"Number","intermittent":"Number","name":"String","class":"String","name:latin":"String","name_en":"String"},"minzoom":4,"maxzoom":14}]}
0:01:14 INF [mbtiles:write] - Starting z0
0:01:14 INF [mbtiles:write] - Finished z0 in 0s cpu:0s avg:0, now starting z1
0:01:14 INF [mbtiles:write] - Finished z1 in 0s cpu:0s avg:0, now starting z2
0:01:14 INF [mbtiles:write] - Finished z2 in 0s cpu:0s avg:0, now starting z3
0:01:14 INF [mbtiles:write] - Finished z3 in 0s cpu:0s avg:0, now starting z4
0:01:14 INF [mbtiles:write] - Finished z4 in 0s cpu:0s avg:0, now starting z5
0:01:14 INF [mbtiles:write] - Finished z5 in 0s cpu:0s avg:0, now starting z6
0:01:14 INF [mbtiles:write] - Finished z6 in 0s cpu:0s avg:0, now starting z7
0:01:14 INF [mbtiles:write] - Finished z7 in 0s cpu:0s avg:0, now starting z8
0:01:15 INF [mbtiles:write] - Finished z8 in 0.9s cpu:2s avg:2, now starting z9
0:01:16 INF [mbtiles:write] - Finished z9 in 1s cpu:2s avg:2, now starting z10
0:01:16 INF [mbtiles:write] - Finished z10 in 0.1s cpu:0.2s avg:2, now starting z11
0:01:17 INF [mbtiles:write] - Finished z11 in 0.8s cpu:2s avg:2, now starting z12
0:01:19 INF [mbtiles:write] - Finished z12 in 2s cpu:4s avg:2, now starting z13
0:01:24 INF [mbtiles] -  features: [ 728k  14%  69k/s ] 268M  tiles: [ 291k  27k/s ] 46M  
    cpus: 2 gc:  8% heap: 1G/4.2G direct: 54M postGC: 875M
    read( 3%) -> (214/217) -> encode(59% 57%) -> (215/216) -> write( 7%)
    last tile: 13/2470/3047 (z13 4%) https://www.openstreetmap.org/#map=13/41.80408/-71.45508
0:01:29 INF [mbtiles:write] - Finished z13 in 10s cpu:20s avg:2, now starting z14
0:01:34 INF [mbtiles] -  features: [ 4.1M  78% 337k/s ] 268M  tiles: [ 2.7M 246k/s ] 355M 
    cpus: 1.8 gc:  1% heap: 2.3G/4.2G direct: 54M postGC: 924M
    read(10%) ->   (0/217) -> encode(43% 43%) -> (215/216) -> write(50%)
    last tile: 14/6269/6855 (z14 56%) https://www.openstreetmap.org/#map=14/28.16888/-42.25342
0:01:37 INF [mbtiles:write] - Finished z14 in 7s cpu:11s avg:1.5
0:01:37 INF [mbtiles] -  features: [ 5.2M 100% 426k/s ] 268M  tiles: [ 4.1M 491k/s ] 514M 
    cpus: 1.4 gc:  0% heap: 3G/4.2G direct: 54M postGC: 924M
    read( -%) ->   (0/217) -> encode( -%  -%) ->   (0/216) -> write( -%)
    last tile: 14/7380/5985 (z14 100%) https://www.openstreetmap.org/#map=14/43.56447/-17.84180
0:01:37 DEB [mbtiles] - Tile stats:
0:01:37 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:37 DEB [mbtiles] - z1 avg:4k max:4k
0:01:37 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:37 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:37 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:37 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:37 DEB [mbtiles] - z6 avg:1.4k max:24k
0:01:37 DEB [mbtiles] - z7 avg:899 max:33k
0:01:37 DEB [mbtiles] - z8 avg:366 max:48k
0:01:37 DEB [mbtiles] - z9 avg:296 max:278k
0:01:37 DEB [mbtiles] - z10 avg:164 max:232k
0:01:37 DEB [mbtiles] - z11 avg:107 max:132k
0:01:37 DEB [mbtiles] - z12 avg:86 max:118k
0:01:37 DEB [mbtiles] - z13 avg:72 max:109k
0:01:37 DEB [mbtiles] - z14 avg:68 max:256k
0:01:37 DEB [mbtiles] - all avg:70 max:0
0:01:37 DEB [mbtiles] -  # features: 5,277,787
0:01:37 DEB [mbtiles] -     # tiles: 4,115,423
0:01:37 INF [mbtiles] - Finished in 23s cpu:43s avg:1.8
0:01:37 INF [mbtiles] -   read    1x(7% 2s wait:19s)
0:01:37 INF [mbtiles] -   encode  2x(45% 10s wait:6s)
0:01:37 INF [mbtiles] -   write   1x(36% 8s wait:12s)
0:01:37 INF - Finished in 1m37s cpu:2m52s gc:3s avg:1.8
0:01:37 INF - FINISHED!
0:01:37 INF - 
0:01:37 INF - ----------------------------------------
0:01:37 INF - 	overall          1m37s cpu:2m52s gc:3s avg:1.8
0:01:37 INF - 	lake_centerlines 1s cpu:3s avg:1.9
0:01:37 INF - 	  read     1x(80% 1s)
0:01:37 INF - 	  process  2x(13% 0.2s wait:1s)
0:01:37 INF - 	  write    1x(0% 0s wait:1s)
0:01:37 INF - 	water_polygons   23s cpu:40s gc:2s avg:1.8
0:01:37 INF - 	  read     1x(61% 14s wait:2s)
0:01:37 INF - 	  process  2x(27% 6s wait:11s)
0:01:37 INF - 	  write    1x(4% 0.9s wait:22s)
0:01:37 INF - 	natural_earth    8s cpu:13s avg:1.6
0:01:37 INF - 	  read     1x(89% 7s sys:1s)
0:01:37 INF - 	  process  2x(20% 2s wait:7s)
0:01:37 INF - 	  write    1x(0% 0s wait:8s)
0:01:37 INF - 	osm_pass1        3s cpu:7s avg:1.9
0:01:37 INF - 	  read     1x(3% 0.1s wait:3s)
0:01:37 INF - 	  parse    1x(62% 2s wait:1s)
0:01:37 INF - 	  process  1x(51% 2s wait:1s)
0:01:37 INF - 	osm_pass2        27s cpu:54s avg:2
0:01:37 INF - 	  read     1x(0% 0s wait:3s done:24s)
0:01:37 INF - 	  process  2x(74% 20s)
0:01:37 INF - 	  write    1x(1% 0.4s wait:27s)
0:01:37 INF - 	boundaries       0s cpu:0.1s avg:1.5
0:01:37 INF - 	sort             3s cpu:4s avg:1.2
0:01:37 INF - 	  worker  2x(44% 1s done:2s)
0:01:37 INF - 	mbtiles          23s cpu:43s avg:1.8
0:01:37 INF - 	  read    1x(7% 2s wait:19s)
0:01:37 INF - 	  encode  2x(45% 10s wait:6s)
0:01:37 INF - 	  write   1x(36% 8s wait:12s)
0:01:37 INF - ----------------------------------------
0:01:37 INF - 	features	268MB
0:01:37 INF - 	mbtiles	514MB
-rw-r--r-- 1 runner docker 55M Mar 23 00:29 run.jar

ℹ️ This Branch Logs f80e6dd

0:00:00 DEB - argument: config=null (path to config file)
0:00:00 DEB - argument: area=rhode island (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet'))
0:00:00 INF - Using in-memory stats
0:00:00 INF [overall] - 
0:00:00 INF [overall] - Starting...
0:00:00 DEB - argument: madvise=true (default value for whether to use linux madvise(random) to improve memory-mapped read performance for temporary storage)
0:00:00 DEB - argument: storage=mmap (default storage type for temporary data, one of [ram, mmap, direct])
0:00:00 DEB - argument: bounds=Env[-74.07 : -17.84, 21.34 : 43.55] (bounds)
0:00:00 DEB - argument: threads=2 (num threads)
0:00:00 DEB - argument: loginterval=10 seconds (time between logs)
0:00:00 DEB - argument: minzoom=0 (minimum zoom level)
0:00:00 DEB - argument: maxzoom=14 (maximum zoom level (limit 14))
0:00:00 DEB - argument: defer_mbtiles_index_creation=false (skip adding index to mbtiles file)
0:00:00 DEB - argument: optimize_db=false (optimize mbtiles after writing)
0:00:00 DEB - argument: emit_tiles_in_order=true (emit tiles in index order)
0:00:00 DEB - argument: force=false (overwriting output file and ignore disk/RAM warnings)
0:00:00 DEB - argument: gzip_temp=false (gzip temporary feature storage (uses more CPU, but less disk space))
0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks)
0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks)
0:00:00 DEB - argument: nodemap_type=sparsearray (type of node location map, one of [noop, sortedtable, sparsearray, array])
0:00:00 DEB - argument: nodemap_storage=mmap (storage for node location map, one of [ram, mmap, direct])
0:00:00 DEB - argument: nodemap_madvise=true (use linux madvise(random) for node locations)
0:00:00 DEB - argument: multipolygon_geometry_storage=mmap (storage for multipolygon geometries, one of [ram, mmap, direct])
0:00:00 DEB - argument: multipolygon_geometry_madvise=false (use linux madvise(random) for temporary multipolygon geometry storage)
0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP)
0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP)
0:00:00 DEB - argument: http_retries=1 (Retries to use when downloading files over HTTP)
0:00:00 DEB - argument: download_chunk_size_mb=100 (Size of file chunks to download in parallel in megabytes)
0:00:00 DEB - argument: download_threads=1 (Number of parallel threads to use when downloading each file)
0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level)
0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level)
0:00:00 DEB - argument: osm_lazy_reads=false (Read OSM blocks from disk in worker threads)
0:00:00 DEB - argument: tmpdir=data/tmp (temp directory)
0:00:00 DEB - argument: only_download=false (download source data then exit)
0:00:00 DEB - argument: download=false (download sources)
0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit)
0:00:00 DEB - argument: fetch_wikidata=false (fetch wikidata translations then continue)
0:00:00 DEB - argument: use_wikidata=true (use wikidata translations)
0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file)
0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path)
0:00:00 DEB - argument: free_lake_centerlines_after_read=false (delete lake_centerlines input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path)
0:00:00 DEB - argument: free_water_polygons_after_read=false (delete water_polygons input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path)
0:00:00 DEB - argument: free_natural_earth_after_read=false (delete natural_earth input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: osm_path=data/sources/rhode_island.osm.pbf (osm OSM input file path)
0:00:00 DEB - argument: free_osm_after_read=false (delete osm input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: mbtiles=data/out.mbtiles (mbtiles output file)
0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names)
0:00:00 DEB - argument: languages=am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh (languages to use)
0:00:00 DEB - argument: only_layers= (Include only certain layers)
0:00:00 DEB - argument: exclude_layers= (Exclude certain layers)
0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries)
0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13)
0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13)
0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways)
0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield))
0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways)
0:00:00 DEB - argument: mbtiles_name=OpenMapTiles ('name' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a> ('attribution' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_version=3.13.0 ('version' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_type=baselayer ('type' attribute for mbtiles metadata)
0:00:00 DEB - argument: help=false (show arguments then exit)
0:00:00 INF - Building BasemapProfile profile into data/out.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   mbtiles: Encode each tile and write to data/out.mbtiles
0:00:00 INF - error loading /home/runner/work/planetiler/planetiler/data/sources/wikidata_names.json: java.nio.file.NoSuchFileException: data/sources/wikidata_names.json
0:00:00 DEB - ✓ 190M storage on / (/dev/root) requested for read phase disk, 32G available
0:00:00 DEB -  - 43M used for temporary node location cache
0:00:00 DEB -  - 3.3M used for temporary multipolygon geometry cache
0:00:00 DEB -  - 143M used for temporary feature storage
0:00:00 DEB - ✓ 215M storage on / (/dev/root) requested for write phase disk, 32G available
0:00:00 DEB -  - 143M used for temporary feature storage
0:00:00 DEB -  - 71M used for mbtiles output
0:00:00 DEB - ✓ 312M JVM heap requested for read phase, 4.2G available
0:00:00 DEB -  - 300M used for sparsearray node location in-memory index
0:00:00 DEB -  - 12M used for temporary profile storage
0:00:00 DEB - ✓ 46M storage on / (/dev/root) requested for read phase, 32G available
0:00:00 DEB -  - 43M used for sparsearray node location cache
0:00:00 DEB -  - 3.3M used for multipolygon way geometries
0:00:00 DEB - ✓ 46M temporary files and 2.9G of free memory for OS to cache them
0:00:00 INF - Using merge sort feature map, chunk size=1000mb workers=2
0:00:02 INF - dataFileCache open start
0:00:02 INF [lake_centerlines] - 
0:00:02 INF [lake_centerlines] - Starting...
0:00:03 INF [lake_centerlines] -  read: [  51k 100%  36k/s ] write: [    0    0/s ] 0    
    cpus: 1.8 gc:  4% heap: 150M/4.2G direct: 73k postGC: 79M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:03 INF [lake_centerlines] - Finished in 1s cpu:3s avg:1.8
0:00:03 INF [lake_centerlines] -   read     1x(78% 1s)
0:00:03 INF [lake_centerlines] -   process  2x(14% 0.2s wait:1s)
0:00:03 INF [lake_centerlines] -   write    1x(0% 0s wait:1s)
0:00:03 INF [water_polygons] - 
0:00:03 INF [water_polygons] - Starting...
0:00:13 INF [water_polygons] -  read: [ 2.2k  16%  224/s ] write: [  59k 5.8k/s ] 4.2M 
    cpus: 1.9 gc:  8% heap: 3.5G/4.2G direct: 54M postGC: 878M
    read(59%) ->  (121/1k) -> process(25% 32%) -> (847/53k) -> write( 0%)
0:00:23 INF [water_polygons] -  read: [   6k  42%  376/s ] write: [ 322k  26k/s ] 16M  
    cpus: 1.6 gc: 11% heap: 1.8G/4.2G direct: 54M postGC: 1.6G
    read(72%) ->    (0/1k) -> process(21% 14%) ->  (1k/53k) -> write( 1%)
0:00:27 INF [water_polygons] -  read: [  14k 100% 2.3k/s ] write: [ 4.3M 1.1M/s ] 186M 
    cpus: 1.6 gc:  6% heap: 2.6G/4.2G direct: 54M postGC: 1.7G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:27 INF [water_polygons] - Finished in 24s cpu:41s gc:2s avg:1.7
0:00:27 INF [water_polygons] -   read     1x(60% 14s wait:2s)
0:00:27 INF [water_polygons] -   process  2x(26% 6s wait:11s)
0:00:27 INF [water_polygons] -   write    1x(4% 0.8s wait:23s)
0:00:27 INF [natural_earth] - unzipping /home/runner/work/planetiler/planetiler/data/sources/natural_earth_vector.sqlite.zip to data/tmp/natearth.sqlite
0:00:36 INF [natural_earth] - 
0:00:36 INF [natural_earth] - Starting...
0:00:45 INF [natural_earth] -  read: [ 349k 100%  44k/s ] write: [  181   22/s ] 186M 
    cpus: 1.7 gc:  0% heap: 2.1G/4.2G direct: 54M postGC: 1.7G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:45 INF [natural_earth] - Finished in 9s cpu:14s avg:1.5
0:00:45 INF [natural_earth] -   read     1x(78% 7s sys:1s done:2s)
0:00:45 INF [natural_earth] -   process  2x(17% 2s wait:7s done:2s)
0:00:45 INF [natural_earth] -   write    1x(0% 0s wait:8s done:2s)
0:00:45 INF [osm_pass1] - 
0:00:45 INF [osm_pass1] - Starting...
0:00:48 INF [osm_pass1:process] - Finished nodes: 4,576,086 (2M/s) in 2s cpu:4s avg:1.7
0:00:48 INF [osm_pass1:process] - Finished ways: 326,826 (462k/s) in 0.7s cpu:1s avg:2
0:00:48 INF [osm_pass1:process] - Finished relations: 7,573 (128k/s) in 0.1s cpu:0.1s avg:1.9
0:00:48 INF [osm_pass1] -  nodes: [ 4.5M 1.4M/s ] 408M  ways: [ 326k 106k/s ] rels: [ 7.5k 2.4k/s ] blocks: [  615  201/s ]
    cpus: 1.8 gc:  1% heap: 959M/4.2G direct: 54M postGC: 856M hppc: 879k
    read( -%) ->     (0/4) -> parse( -%) ->     (0/4) -> process( -%)
0:00:48 DEB [osm_pass1] - Processed 615 blocks:
0:00:48 DEB [osm_pass1] -   nodes: 4,576,086 (2M/s) in 2s cpu:4s avg:1.7
0:00:48 DEB [osm_pass1] -   ways: 326,826 (462k/s) in 0.7s cpu:1s avg:2
0:00:48 DEB [osm_pass1] -   relations: 7,573 (128k/s) in 0.1s cpu:0.1s avg:1.9
0:00:48 INF [osm_pass1] - Finished in 3s cpu:5s avg:1.8
0:00:48 INF [osm_pass1] -   read     1x(3% 0.1s wait:3s)
0:00:48 INF [osm_pass1] -   parse    1x(63% 2s)
0:00:48 INF [osm_pass1] -   process  1x(49% 1s wait:1s)
0:00:48 INF [osm_pass2] - 
0:00:48 INF [osm_pass2] - Starting...
0:00:52 DEB [osm_pass2:process] - Sorting long long multimap...
0:00:52 INF [osm_pass2:process] - Finished nodes: 4,576,086 (1.2M/s) in 4s cpu:7s avg:2
0:00:52 DEB [osm_pass2:process] - Sorted long long multimap 0s cpu:0s avg:1.5
0:00:52 WAR [osm_pass2:process] - No GB polygon for inferring route network types
0:00:58 INF [osm_pass2] -  nodes: [ 4.5M 100% 457k/s ] 408M  ways: [ 178k  55%  17k/s ] rels: [    0   0%    0/s ] features: [ 4.9M  62k/s ] 230M  blocks: [  594  97%   59/s ]
    cpus: 2 gc:  1% heap: 1G/4.2G direct: 55M postGC: 866M relInfo: 820k mpGeoms: 649k 
    read( 0%) ->   (11/13) -> process(68% 68%) -> (1.1k/53k) -> write( 2%)
0:01:05 INF [osm_pass2:process] - Finished ways: 326,826 (24k/s) in 13s cpu:26s avg:2
0:01:08 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 408M  ways: [ 326k 100%  14k/s ] rels: [   3k  40%  303/s ] features: [ 5.2M  30k/s ] 257M  blocks: [  614 100%    1/s ]
    cpus: 2 gc:  1% heap: 1.3G/4.2G direct: 54M postGC: 856M relInfo: 820k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process(79% 80%) ->  (1k/53k) -> write( 1%)
0:01:12 INF [osm_pass2:process] - Finished relations: 7,573 (1.1k/s) in 7s cpu:13s avg:2
0:01:14 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 408M  ways: [ 326k 100%    0/s ] rels: [ 7.5k 100%  751/s ] features: [ 5.2M 3.7k/s ] 268M  blocks: [  615 100%   <1/s ]
    cpus: 2 gc:  0% heap: 1.7G/4.2G direct: 54M postGC: 850M relInfo: 820k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:01:14 DEB [osm_pass2] - Processed 615 blocks:
0:01:14 DEB [osm_pass2] -   nodes: 4,576,086 (1.2M/s) in 4s cpu:7s avg:2
0:01:14 DEB [osm_pass2] -   ways: 326,826 (24k/s) in 13s cpu:26s avg:2
0:01:14 DEB [osm_pass2] -   relations: 7,573 (1.1k/s) in 7s cpu:13s avg:2
0:01:14 INF [osm_pass2] - Finished in 26s cpu:52s avg:2
0:01:14 INF [osm_pass2] -   read     1x(0% 0s wait:13s done:13s)
0:01:14 INF [osm_pass2] -   process  2x(76% 20s)
0:01:14 INF [osm_pass2] -   write    1x(1% 0.4s wait:26s)
0:01:14 INF [boundaries] - 
0:01:14 INF [boundaries] - Starting...
0:01:14 INF [boundaries] - Creating polygons for 1 boundaries
0:01:14 WAR [boundaries] - Unable to form closed polygon for OSM relation 148838 (likely missing edges)
0:01:14 INF [boundaries] - Finished creating 0 country polygons
0:01:14 INF [boundaries] - Finished in 0s cpu:0.1s avg:1.5
0:01:14 INF - Deleting node.db to make room for output file
0:01:14 INF [sort] - 
0:01:14 INF [sort] - Starting...
0:01:19 INF [sort] -  chunks: [   1 /   1 100% ] 268M 
    cpus: 1.1 gc:  0% heap: 2.2G/4.2G direct: 54M postGC: 850M
    ->     (0/4) -> worker( -%  -%)
0:01:19 INF [sort] - Finished in 4s cpu:4s avg:1.1
0:01:19 INF [sort] -   worker  2x(38% 2s done:2s)
0:01:19 INF - read:1s write:1s sort:1s
0:01:19 INF [mbtiles] - 
0:01:19 INF [mbtiles] - Starting...
0:01:19 DEB [mbtiles:write] - Execute mbtiles: create table metadata (name text, value text);
0:01:19 DEB [mbtiles:write] - Execute mbtiles: create unique index name on metadata (name);
0:01:19 DEB [mbtiles:write] - Execute mbtiles: create table tiles (zoom_level integer, tile_column integer, tile_row, tile_data blob);
0:01:19 DEB [mbtiles:write] - Execute mbtiles: create unique index tile_index on tiles (zoom_level, tile_column, tile_row)
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: name=OpenMapTiles
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: format=pbf
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: version=3.13.0
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: type=baselayer
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: bounds=-74.07,21.34,-17.84,43.55
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: center=-45.955,32.445,3
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: minzoom=0
0:01:19 DEB [mbtiles:write] - Set mbtiles metadata: maxzoom=14
0:01:20 DEB [mbtiles:write] - Set mbtiles metadata: json={"vector_layers":[{"id":"aerodrome_label","fields":{"name_int":"String","iata":"String","ele_ft":"Number","name_de":"String","name":"String","icao":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":10,"maxzoom":14},{"id":"aeroway","fields":{"ref":"String","class":"String"},"minzoom":10,"maxzoom":14},{"id":"boundary","fields":{"disputed":"Number","admin_level":"Number","maritime":"Number","disputed_name":"String"},"minzoom":0,"maxzoom":14},{"id":"building","fields":{"colour":"String","render_height":"Number","render_min_height":"Number"},"minzoom":13,"maxzoom":14},{"id":"housenumber","fields":{"housenumber":"String"},"minzoom":14,"maxzoom":14},{"id":"landcover","fields":{"subclass":"String","class":"String","_numpoints":"Number"},"minzoom":8,"maxzoom":14},{"id":"landuse","fields":{"class":"String"},"minzoom":4,"maxzoom":14},{"id":"mountain_peak","fields":{"name_int":"String","customary_ft":"Number","ele_ft":"Number","name_de":"String","name":"String","rank":"Number","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":7,"maxzoom":14},{"id":"park","fields":{"name_int":"String","name_de":"String","name":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"place","fields":{"name:fy":"String","name_int":"String","capital":"Number","name:uk":"String","name:pl":"String","name:nl":"String","name:be":"String","name:ru":"String","name:ko":"String","name_de":"String","name":"String","rank":"Number","name:en":"String","name:eo":"String","class":"String","name:hu":"String","name:ta":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":2,"maxzoom":14},{"id":"poi","fields":{"name_int":"String","level":"Number","name:nonlatin":"String","layer":"Number","name_de":"String","name":"String","subclass":"String","indoor":"Number","name:en":"String","class":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":12,"maxzoom":14},{"id":"transportation","fields":{"brunnel":"String","access":"String","expressway":"Number","bicycle":"String","surface":"String","level":"Number","ramp":"Number","mtb_scale":"String","toll":"Number","layer":"Number","oneway":"Number","network":"String","horse":"String","service":"String","subclass":"String","class":"String","foot":"String"},"minzoom":4,"maxzoom":14},{"id":"transportation_name","fields":{"name_int":"String","name:nonlatin":"String","route_4":"String","route_3":"String","route_2":"String","route_1":"String","layer":"Number","network":"String","ref":"String","name_de":"String","name":"String","subclass":"String","ref_length":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"water","fields":{"intermittent":"Number","class":"String"},"minzoom":0,"maxzoom":14},{"id":"water_name","fields":{"name_int":"String","name:nonlatin":"String","name_de":"String","name":"String","intermittent":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":9,"maxzoom":14},{"id":"waterway","fields":{"name_int":"String","brunnel":"String","name_de":"String","_relid":"Number","intermittent":"Number","name":"String","class":"String","name:latin":"String","name_en":"String"},"minzoom":4,"maxzoom":14}]}
0:01:20 INF [mbtiles:write] - Starting z0
0:01:20 INF [mbtiles:write] - Finished z0 in 0s cpu:0s avg:12.2, now starting z1
0:01:20 INF [mbtiles:write] - Finished z1 in 0s cpu:0s avg:0, now starting z2
0:01:20 INF [mbtiles:write] - Finished z2 in 0s cpu:0s avg:0, now starting z3
0:01:20 INF [mbtiles:write] - Finished z3 in 0s cpu:0s avg:0, now starting z4
0:01:20 INF [mbtiles:write] - Finished z4 in 0s cpu:0s avg:0, now starting z5
0:01:20 INF [mbtiles:write] - Finished z5 in 0s cpu:0s avg:0, now starting z6
0:01:20 INF [mbtiles:write] - Finished z6 in 0s cpu:0s avg:52.9, now starting z7
0:01:20 INF [mbtiles:write] - Finished z7 in 0.1s cpu:0.1s avg:2, now starting z8
0:01:21 INF [mbtiles:write] - Finished z8 in 0.8s cpu:2s avg:2, now starting z9
0:01:22 INF [mbtiles:write] - Finished z9 in 1s cpu:2s avg:2, now starting z10
0:01:22 INF [mbtiles:write] - Finished z10 in 0.1s cpu:0.2s avg:2, now starting z11
0:01:22 INF [mbtiles:write] - Finished z11 in 0.7s cpu:1s avg:2, now starting z12
0:01:25 INF [mbtiles:write] - Finished z12 in 2s cpu:4s avg:2, now starting z13
0:01:29 INF [mbtiles] -  features: [ 728k  14%  72k/s ] 268M  tiles: [ 291k  29k/s ] 46M  
    cpus: 2 gc:  4% heap: 2.4G/4.2G direct: 54M postGC: 890M
    read( 3%) -> (214/217) -> encode(57% 58%) -> (215/216) -> write( 8%)
    last tile: 13/2470/3047 (z13 4%) https://www.openstreetmap.org/#map=13/41.80408/-71.45508
0:01:34 INF [mbtiles:write] - Finished z13 in 10s cpu:19s avg:2, now starting z14
0:01:39 INF [mbtiles] -  features: [ 4.1M  80% 346k/s ] 268M  tiles: [ 2.8M 256k/s ] 366M 
    cpus: 1.7 gc:  1% heap: 1.1G/4.2G direct: 54M postGC: 944M
    read(10%) ->   (0/217) -> encode(40% 42%) -> (214/216) -> write(53%)
    last tile: 14/6341/6191 (z14 59%) https://www.openstreetmap.org/#map=14/40.19566/-40.67139
0:01:42 INF [mbtiles:write] - Finished z14 in 8s cpu:11s avg:1.5
0:01:42 INF [mbtiles] -  features: [ 5.2M 100% 396k/s ] 268M  tiles: [ 4.1M 463k/s ] 514M 
    cpus: 1.3 gc:  0% heap: 1.7G/4.2G direct: 54M postGC: 944M
    read( -%) ->   (0/217) -> encode( -%  -%) ->   (0/216) -> write( -%)
    last tile: 14/7380/5985 (z14 100%) https://www.openstreetmap.org/#map=14/43.56447/-17.84180
0:01:42 DEB [mbtiles] - Tile stats:
0:01:42 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:42 DEB [mbtiles] - z1 avg:4k max:4k
0:01:42 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:42 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:42 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:42 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:42 DEB [mbtiles] - z6 avg:1.4k max:24k
0:01:42 DEB [mbtiles] - z7 avg:899 max:33k
0:01:42 DEB [mbtiles] - z8 avg:366 max:48k
0:01:42 DEB [mbtiles] - z9 avg:296 max:278k
0:01:42 DEB [mbtiles] - z10 avg:164 max:232k
0:01:42 DEB [mbtiles] - z11 avg:107 max:132k
0:01:42 DEB [mbtiles] - z12 avg:86 max:118k
0:01:42 DEB [mbtiles] - z13 avg:72 max:109k
0:01:42 DEB [mbtiles] - z14 avg:68 max:256k
0:01:42 DEB [mbtiles] - all avg:70 max:0
0:01:42 DEB [mbtiles] -  # features: 5,277,787
0:01:42 DEB [mbtiles] -     # tiles: 4,115,423
0:01:42 INF [mbtiles] - Finished in 23s cpu:42s avg:1.8
0:01:42 INF [mbtiles] -   read    1x(8% 2s wait:19s)
0:01:42 INF [mbtiles] -   encode  2x(44% 10s wait:6s)
0:01:42 INF [mbtiles] -   write   1x(37% 9s sys:1s wait:12s)
0:01:42 INF - Finished in 1m42s cpu:2m50s gc:3s avg:1.7
0:01:42 INF - FINISHED!
0:01:42 INF - 
0:01:42 INF - ----------------------------------------
0:01:42 INF - 	overall          1m42s cpu:2m50s gc:3s avg:1.7
0:01:42 INF - 	lake_centerlines 1s cpu:3s avg:1.8
0:01:42 INF - 	  read     1x(78% 1s)
0:01:42 INF - 	  process  2x(14% 0.2s wait:1s)
0:01:42 INF - 	  write    1x(0% 0s wait:1s)
0:01:42 INF - 	water_polygons   24s cpu:41s gc:2s avg:1.7
0:01:42 INF - 	  read     1x(60% 14s wait:2s)
0:01:42 INF - 	  process  2x(26% 6s wait:11s)
0:01:42 INF - 	  write    1x(4% 0.8s wait:23s)
0:01:42 INF - 	natural_earth    9s cpu:14s avg:1.5
0:01:42 INF - 	  read     1x(78% 7s sys:1s done:2s)
0:01:42 INF - 	  process  2x(17% 2s wait:7s done:2s)
0:01:42 INF - 	  write    1x(0% 0s wait:8s done:2s)
0:01:42 INF - 	osm_pass1        3s cpu:5s avg:1.8
0:01:42 INF - 	  read     1x(3% 0.1s wait:3s)
0:01:42 INF - 	  parse    1x(63% 2s)
0:01:42 INF - 	  process  1x(49% 1s wait:1s)
0:01:42 INF - 	osm_pass2        26s cpu:52s avg:2
0:01:42 INF - 	  read     1x(0% 0s wait:13s done:13s)
0:01:42 INF - 	  process  2x(76% 20s)
0:01:42 INF - 	  write    1x(1% 0.4s wait:26s)
0:01:42 INF - 	boundaries       0s cpu:0.1s avg:1.5
0:01:42 INF - 	sort             4s cpu:4s avg:1.1
0:01:42 INF - 	  worker  2x(38% 2s done:2s)
0:01:42 INF - 	mbtiles          23s cpu:42s avg:1.8
0:01:42 INF - 	  read    1x(8% 2s wait:19s)
0:01:42 INF - 	  encode  2x(44% 10s wait:6s)
0:01:42 INF - 	  write   1x(37% 9s sys:1s wait:12s)
0:01:42 INF - ----------------------------------------
0:01:42 INF - 	features	268MB
0:01:42 INF - 	mbtiles	514MB
-rw-r--r-- 1 runner docker 55M Mar 23 00:27 run.jar

msbarry · 2022-03-23T00:18:48Z

Some stats from the -Xmx32g on a 64GB machine:

0:07:27 DEB [osm_pass1] - Processed 18,767 blocks:
0:07:27 DEB [osm_pass1] -   nodes: 7,546,573,682 (32M/s) in 3m55s cpu:52m15s gc:5s avg:13.3
0:07:27 DEB [osm_pass1] -   ways: 841,994,745 (9.6M/s) in 1m28s cpu:21m1s gc:5s avg:14.4
0:07:27 DEB [osm_pass1] -   relations: 9,718,612 (852k/s) in 11s cpu:59s avg:5.2
...
1:43:11 DEB [osm_pass2] -   nodes: 7,546,573,682 (11M/s) in 11m22s cpu:2h45m5s gc:17s avg:14.5
1:43:11 DEB [osm_pass2] -   ways: 841,994,745 (200k/s) in 1h9m58s cpu:13h8m7s gc:2m53s avg:11.3
1:43:11 DEB [osm_pass2] -   relations: 9,718,612 (11k/s) in 14m26s cpu:2h57m49s gc:17s avg:12.3
...
2:19:17 INF - 	overall          2h19m18s cpu:26h50m15s gc:4m47s avg:11.6
2:19:17 INF - 	lake_centerlines 2s cpu:4s avg:2.7
2:19:17 INF - 	  read     1x(70% 1s)
2:19:17 INF - 	  process 16x(4% 0.1s wait:1s)
2:19:17 INF - 	  write    1x(0% 0s wait:2s)
2:19:17 INF - 	water_polygons   1m34s cpu:17m49s gc:3s avg:11.3
2:19:17 INF - 	  read     1x(18% 17s wait:1m8s done:5s)
2:19:17 INF - 	  process 16x(61% 57s wait:28s)
2:19:17 INF - 	  write    1x(50% 47s sys:8s wait:46s)
2:19:17 INF - 	natural_earth    10s cpu:27s avg:2.8
2:19:17 INF - 	  read     1x(72% 7s wait:2s)
2:19:17 INF - 	  process 16x(6% 0.5s wait:8s)
2:19:17 INF - 	  write    1x(0% 0s wait:9s)
2:19:17 INF - 	osm_pass1        5m35s cpu:1h14m31s gc:11s avg:13.3
2:19:17 INF - 	  read     1x(22% 1m12s sys:46s wait:2m37s done:4s)
2:19:17 INF - 	  process 15x(84% 4m43s sys:6s block:11s wait:19s)
2:19:17 INF - 	osm_pass2        1h35m44s cpu:18h50m49s gc:3m27s avg:11.8
2:19:17 INF - 	  read     1x(1% 1m20s sys:55s wait:1h32m9s done:1m28s)
2:19:17 INF - 	  process 15x(74% 1h10m28s sys:2m12s block:28s wait:5s)
2:19:17 INF - 	  write    1x(21% 19m47s sys:3m40s wait:1h14m27s)
2:19:17 INF - 	boundaries       15s cpu:19s avg:1.3
2:19:17 INF - 	sort             6m44s cpu:1h7m35s gc:34s avg:10
2:19:17 INF - 	  worker 16x(44% 2m59s sys:23s wait:1m12s done:9s)
2:19:17 INF - 	mbtiles          29m6s cpu:5h18m26s gc:34s avg:10.9
2:19:17 INF - 	  read    1x(81% 23m41s sys:3m12s wait:2m23s done:4s)
2:19:17 INF - 	  encode 16x(58% 16m51s sys:3s wait:10m40s done:4s)
2:19:17 INF - 	  write   1x(54% 15m43s sys:2m19s wait:12m2s)

And it took 5.3 CPU-hours fetching way geometries

From https://observablehq.com/@msbarry/planetiler-log-visualizer

msbarry · 2022-03-23T00:20:43Z

And some stats from the -Xmx206 on a 32GB machine:

0:09:34 DEB [osm_pass1] - Processed 18,767 blocks:
0:09:34 DEB [osm_pass1] -   nodes: 7,546,573,682 (22M/s) in 5m30s cpu:48m51s gc:18s avg:8.9
0:09:34 DEB [osm_pass1] -   ways: 841,994,745 (7.6M/s) in 1m51s cpu:25m8s gc:10s avg:13.6
0:09:34 DEB [osm_pass1] -   relations: 9,718,612 (855k/s) in 11s cpu:1m14s avg:6.5
...
2:14:22 DEB [osm_pass2] - Processed 18,767 blocks:
2:14:22 DEB [osm_pass2] -   nodes: 7,546,573,682 (9M/s) in 13m58s cpu:3h22m42s gc:22s avg:14.5
2:14:22 DEB [osm_pass2] -   ways: 841,994,745 (151k/s) in 1h32m20s cpu:12h35m29s gc:2m56s avg:8.2
2:14:22 DEB [osm_pass2] -   relations: 9,718,612 (8.7k/s) in 18m32s cpu:2h52m54s gc:16s avg:9.3
...
2:58:08 INF - 	overall          2h58m8s cpu:26h39m53s gc:5m21s avg:9
2:58:08 INF - 	lake_centerlines 1s cpu:5s avg:3.5
2:58:08 INF - 	  read     1x(75% 1s)
2:58:08 INF - 	  process 16x(5% 0.1s wait:1s)
2:58:08 INF - 	  write    1x(0% 0s wait:1s)
2:58:08 INF - 	water_polygons   1m44s cpu:20m30s gc:3s avg:11.9
2:58:08 INF - 	  read     1x(18% 18s wait:1m12s done:5s)
2:58:08 INF - 	  process 16x(64% 1m7s wait:27s)
2:58:08 INF - 	  write    1x(45% 46s sys:8s wait:56s)
2:58:08 INF - 	natural_earth    8s cpu:24s avg:2.9
2:58:08 INF - 	  read     1x(82% 7s)
2:58:08 INF - 	  process 16x(7% 0.6s wait:7s)
2:58:08 INF - 	  write    1x(0% 0s wait:8s)
2:58:08 INF - 	osm_pass1        7m32s cpu:1h15m20s gc:28s avg:10
2:58:08 INF - 	  read     1x(15% 1m6s sys:44s wait:1m36s done:3s)
2:58:08 INF - 	  process 15x(60% 4m31s sys:9s block:12s wait:2m9s)
2:58:08 INF - 	osm_pass2        2h4m49s cpu:18h50m55s gc:3m35s avg:9.1
2:58:08 INF - 	  read     1x(1% 1m11s sys:49s wait:1h59m35s done:1m59s)
2:58:08 INF - 	  process 15x(56% 1h10m21s sys:3m3s block:27s wait:6s)
2:58:08 INF - 	  write    1x(15% 18m47s sys:3m22s wait:1h44m40s)
2:58:08 INF - 	boundaries       14s cpu:19s avg:1.3
2:58:08 INF - 	sort             14m25s cpu:51m5s gc:39s avg:3.5
2:58:08 INF - 	  worker 16x(15% 2m8s sys:19s wait:5m12s done:11s)
2:58:08 INF - 	mbtiles          29m6s cpu:5h21m2s gc:35s avg:11
2:58:08 INF - 	  read    1x(77% 22m32s sys:3m6s wait:2m50s done:2s)
2:58:08 INF - 	  encode 16x(59% 17m6s sys:3s wait:10m32s done:2s)
2:58:08 INF - 	  write   1x(53% 15m23s sys:2m14s wait:12m24s)

And it took 12.1 CPU-hours fetching way geometries

From https://observablehq.com/@msbarry/planetiler-log-visualizer

…onthegomap#141)

msbarry added 2 commits March 20, 2022 20:49

offload way geometries to disk

598becd

move multipolygon membership storage to disk

d668c8b

msbarry added 2 commits March 22, 2022 09:13

clean up long long multimap interface

4e9a780

control via command line flags, estimate resource usage

43ad62d

rearrange code

ff5d867

msbarry merged commit e20b41c into main Mar 23, 2022

msbarry deleted the mmap-way-geoms branch March 23, 2022 00:35

mmc1718 pushed a commit to nmr8acme/planetiler that referenced this pull request Jan 24, 2023

Offload multipolygon storage to disk (reduce memory usage by 10-15gb) (…

699bfa6

…onthegomap#141)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Offload multipolygon storage to disk (reduce memory usage by 10-15gb) #141

Offload multipolygon storage to disk (reduce memory usage by 10-15gb) #141

msbarry commented Mar 22, 2022 •

edited

github-actions bot commented Mar 22, 2022 •

edited

msbarry commented Mar 23, 2022

msbarry commented Mar 23, 2022 •

edited

Offload multipolygon storage to disk (reduce memory usage by 10-15gb) #141

Offload multipolygon storage to disk (reduce memory usage by 10-15gb) #141

Conversation

msbarry commented Mar 22, 2022 • edited

github-actions bot commented Mar 22, 2022 • edited

msbarry commented Mar 23, 2022

msbarry commented Mar 23, 2022 • edited

msbarry commented Mar 22, 2022 •

edited

github-actions bot commented Mar 22, 2022 •

edited

msbarry commented Mar 23, 2022 •

edited