TileCoord supports up to zoom 15 using alternate ordering #266

bdon · 2022-06-08T04:01:07Z

This is a rough idea right now (don't merge yet) to explore alternate TileCoord encodings. it deserves some better tests for edge cases as right now it's blindly ported from C++ to java

The current TileCoord uses a fixed 14 bits for X and Y and the remaining bits for Z. This instead defines a mapping from a 32-bit unsigned encoded integer to X,Y,Z like this:

Tile 0 is (0,0) on z0
Tile 1-4 are positions on the z1 Hilbert curve; each side of the curve is length 2, so tile 1 is (0,0), tile 2 is (0,1), tile 3 is (1,1) and tile 4 is (1,0)
Tile 5-20 are positions on the z2 Hilbert curve
etc...

Drawbacks:

decoding/encoding might more expensive
does not correspond to index insertion order in MBTiles, but there may be other ways to mitigate this

Potential benefits of doing it this way:

support z15 without widening the TileCoord encoding to 64 bits
If features are ordered this way and memory-mapped on disk, reads should have better locality for 2D queries vs. striped Z/X/Y ordering
for some other tools I'm developing this would aid interoperability, such as a bitmap format for sparse ocean representation or a future ordering requirement for PMTiles

github-actions · 2022-06-08T04:08:12Z

Base 4b7a601 This Branch 5ab10cb

0:01:59 DEB [mbtiles] - Tile stats:
0:01:59 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:59 DEB [mbtiles] - z1 avg:4k max:4k
0:01:59 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:59 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:59 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:59 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:59 DEB [mbtiles] - z6 avg:973 max:22k
0:01:59 DEB [mbtiles] - z7 avg:751 max:55k
0:01:59 DEB [mbtiles] - z8 avg:394 max:112k
0:01:59 DEB [mbtiles] - z9 avg:270 max:278k
0:01:59 DEB [mbtiles] - z10 avg:154 max:233k
0:01:59 DEB [mbtiles] - z11 avg:102 max:132k
0:01:59 DEB [mbtiles] - z12 avg:83 max:119k
0:01:59 DEB [mbtiles] - z13 avg:71 max:109k
0:01:59 DEB [mbtiles] - z14 avg:67 max:266k
0:01:59 DEB [mbtiles] - all avg:69 max:0
0:01:59 DEB [mbtiles] -  # features: 5,293,577
0:01:59 DEB [mbtiles] -     # tiles: 4,115,450
0:01:59 INF [mbtiles] - Finished in 33s cpu:1m3s avg:1.9
0:01:59 INF [mbtiles] -   read    1x(2% 0.7s wait:31s)
0:01:59 INF [mbtiles] -   encode  2x(56% 18s wait:3s)
0:01:59 INF [mbtiles] -   write   1x(25% 8s wait:23s)
0:01:59 INF - Finished in 2m cpu:3m35s gc:4s avg:1.8
0:01:59 INF - FINISHED!
0:01:59 INF - 
0:01:59 INF - ----------------------------------------
0:01:59 INF - 	overall          2m cpu:3m35s gc:4s avg:1.8
0:01:59 INF - 	lake_centerlines 2s cpu:4s avg:1.9
0:01:59 INF - 	  read     1x(84% 2s)
0:01:59 INF - 	  process  2x(13% 0.3s wait:2s)
0:01:59 INF - 	  write    1x(0% 0s wait:2s)
0:01:59 INF - 	water_polygons   27s cpu:48s gc:2s avg:1.8
0:01:59 INF - 	  read     1x(60% 16s wait:3s)
0:01:59 INF - 	  process  2x(29% 8s wait:13s)
0:01:59 INF - 	  write    1x(2% 0.6s wait:26s)
0:01:59 INF - 	natural_earth    10s cpu:16s avg:1.5
0:01:59 INF - 	  read     1x(86% 9s sys:1s)
0:01:59 INF - 	  process  2x(18% 2s wait:9s)
0:01:59 INF - 	  write    1x(0% 0s wait:10s)
0:01:59 INF - 	osm_pass1        4s cpu:7s avg:1.8
0:01:59 INF - 	  read     1x(2% 0.1s wait:4s)
0:01:59 INF - 	  parse    1x(64% 3s)
0:01:59 INF - 	  process  1x(48% 2s wait:2s)
0:01:59 INF - 	osm_pass2        31s cpu:1m2s avg:2
0:01:59 INF - 	  read     1x(0% 0s wait:15s done:17s)
0:01:59 INF - 	  process  2x(76% 24s)
0:01:59 INF - 	  write    1x(1% 0.4s wait:31s)
0:01:59 INF - 	boundaries       0s cpu:0.1s avg:1.8
0:01:59 INF - 	sort             3s cpu:3s avg:1.2
0:01:59 INF - 	  worker  1x(75% 2s)
0:01:59 INF - 	mbtiles          33s cpu:1m3s avg:1.9
0:01:59 INF - 	  read    1x(2% 0.7s wait:31s)
0:01:59 INF - 	  encode  2x(56% 18s wait:3s)
0:01:59 INF - 	  write   1x(25% 8s wait:23s)
0:01:59 INF - ----------------------------------------
0:01:59 INF - 	features	270MB
0:01:59 INF - 	mbtiles	108MB
-rw-r--r-- 1 runner docker 55M Jul 11 07:00 run.jar

0:02:04 DEB [mbtiles] - Tile stats:
0:02:04 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:02:04 DEB [mbtiles] - z1 avg:4k max:4k
0:02:04 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:02:04 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:02:04 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:02:04 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:02:04 DEB [mbtiles] - z6 avg:973 max:22k
0:02:04 DEB [mbtiles] - z7 avg:751 max:55k
0:02:04 DEB [mbtiles] - z8 avg:394 max:112k
0:02:04 DEB [mbtiles] - z9 avg:270 max:278k
0:02:04 DEB [mbtiles] - z10 avg:154 max:233k
0:02:04 DEB [mbtiles] - z11 avg:102 max:132k
0:02:04 DEB [mbtiles] - z12 avg:83 max:119k
0:02:04 DEB [mbtiles] - z13 avg:71 max:109k
0:02:04 DEB [mbtiles] - z14 avg:67 max:266k
0:02:04 DEB [mbtiles] - all avg:69 max:0
0:02:04 DEB [mbtiles] -  # features: 5,293,577
0:02:04 DEB [mbtiles] -     # tiles: 4,115,450
0:02:04 INF [mbtiles] - Finished in 32s cpu:1m gc:2s avg:1.9
0:02:04 INF [mbtiles] -   read    1x(3% 0.9s wait:29s)
0:02:04 INF [mbtiles] -   encode  2x(54% 17s wait:4s done:1s)
0:02:04 INF [mbtiles] -   write   1x(25% 8s wait:21s)
0:02:04 INF - Finished in 2m4s cpu:3m34s gc:5s avg:1.7
0:02:04 INF - FINISHED!
0:02:04 INF - 
0:02:04 INF - ----------------------------------------
0:02:04 INF - 	overall          2m4s cpu:3m34s gc:5s avg:1.7
0:02:04 INF - 	lake_centerlines 2s cpu:4s avg:1.8
0:02:04 INF - 	  read     1x(77% 2s)
0:02:04 INF - 	  process  2x(11% 0.3s wait:2s)
0:02:04 INF - 	  write    1x(0% 0s wait:2s)
0:02:04 INF - 	water_polygons   28s cpu:48s gc:3s avg:1.7
0:02:04 INF - 	  read     1x(58% 16s wait:3s)
0:02:04 INF - 	  process  2x(28% 8s wait:13s)
0:02:04 INF - 	  write    1x(2% 0.6s wait:27s)
0:02:04 INF - 	natural_earth    10s cpu:16s avg:1.6
0:02:04 INF - 	  read     1x(87% 9s sys:1s)
0:02:04 INF - 	  process  2x(20% 2s wait:9s)
0:02:04 INF - 	  write    1x(0% 0s wait:10s)
0:02:04 INF - 	osm_pass1        5s cpu:7s avg:1.5
0:02:04 INF - 	  read     1x(3% 0.1s wait:4s)
0:02:04 INF - 	  parse    1x(53% 2s wait:1s)
0:02:04 INF - 	  process  1x(35% 2s wait:2s)
0:02:04 INF - 	osm_pass2        32s cpu:1m4s avg:2
0:02:04 INF - 	  read     1x(0% 0.1s wait:16s done:17s)
0:02:04 INF - 	  process  2x(77% 25s)
0:02:04 INF - 	  write    1x(1% 0.4s wait:32s)
0:02:04 INF - 	boundaries       0s cpu:0.1s avg:1.6
0:02:04 INF - 	sort             3s cpu:3s avg:1.2
0:02:04 INF - 	  worker  1x(74% 2s)
0:02:04 INF - 	mbtiles          32s cpu:1m gc:2s avg:1.9
0:02:04 INF - 	  read    1x(3% 0.9s wait:29s)
0:02:04 INF - 	  encode  2x(54% 17s wait:4s done:1s)
0:02:04 INF - 	  write   1x(25% 8s wait:21s)
0:02:04 INF - ----------------------------------------
0:02:04 INF - 	features	270MB
0:02:04 INF - 	mbtiles	108MB
-rw-r--r-- 1 runner docker 55M Jul 11 06:58 run.jar

https://github.com/onthegomap/planetiler/actions/runs/2647800454

ℹ️ Base Logs 4b7a601

0:00:00 DEB - argument: config=null (path to config file)
0:00:00 DEB - argument: area=rhode island (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet'))
0:00:00 INF - Using in-memory stats
0:00:00 INF [overall] - 
0:00:00 INF [overall] - Starting...
0:00:00 DEB - argument: madvise=true (default value for whether to use linux madvise(random) to improve memory-mapped read performance for temporary storage)
0:00:00 DEB - argument: storage=mmap (default storage type for temporary data, one of [ram, mmap, direct])
0:00:00 DEB - argument: threads=2 (num threads)
0:00:00 DEB - argument: write_threads=1 (number of threads to use when writing temp features)
0:00:00 DEB - argument: process_threads=2 (number of threads to use when processing input features)
0:00:00 DEB - argument: bounds=Env[-74.07 : -17.84, 21.34 : 43.55] (bounds)
0:00:00 DEB - argument: feature_read_threads=1 (number of threads to use when reading features at tile write time)
0:00:00 DEB - argument: loginterval=10 seconds (time between logs)
0:00:00 DEB - argument: minzoom=0 (minimum zoom level)
0:00:00 DEB - argument: maxzoom=14 (maximum zoom level (limit 14))
0:00:00 DEB - argument: render_maxzoom=14 (maximum rendering zoom level (limit 14))
0:00:00 DEB - argument: skip_mbtiles_index_creation=false (skip adding index to mbtiles file)
0:00:00 DEB - argument: optimize_db=false (optimize mbtiles after writing)
0:00:00 DEB - argument: emit_tiles_in_order=true (emit tiles in index order)
0:00:00 DEB - argument: force=false (overwriting output file and ignore disk/RAM warnings)
0:00:00 DEB - argument: gzip_temp=false (gzip temporary feature storage (uses more CPU, but less disk space))
0:00:00 DEB - argument: mmap_temp=true (use memory-mapped IO for temp feature files)
0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks)
0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks)
0:00:00 DEB - argument: nodemap_type=sparsearray (type of node location map, one of [noop, sortedtable, sparsearray, array])
0:00:00 DEB - argument: nodemap_storage=mmap (storage for node location map, one of [ram, mmap, direct])
0:00:00 DEB - argument: nodemap_madvise=true (use linux madvise(random) for node locations)
0:00:00 DEB - argument: multipolygon_geometry_storage=mmap (storage for multipolygon geometries, one of [ram, mmap, direct])
0:00:00 DEB - argument: multipolygon_geometry_madvise=true (use linux madvise(random) for temporary multipolygon geometry storage)
0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP)
0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP)
0:00:00 DEB - argument: http_retries=1 (Retries to use when downloading files over HTTP)
0:00:00 DEB - argument: download_chunk_size_mb=100 (Size of file chunks to download in parallel in megabytes)
0:00:00 DEB - argument: download_threads=1 (Number of parallel threads to use when downloading each file)
0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level)
0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level)
0:00:00 DEB - argument: osm_lazy_reads=false (Read OSM blocks from disk in worker threads)
0:00:00 DEB - argument: compact_db=true (Reduce the DB size by separating and deduping the tile data)
0:00:00 DEB - argument: skip_filled_tiles=false (Skip writing tiles containing only polygon fills to the output)
0:00:00 DEB - argument: tmpdir=data/tmp (temp directory)
0:00:00 DEB - argument: only_download=false (download source data then exit)
0:00:00 DEB - argument: download=false (download sources)
0:00:00 DEB - argument: temp_nodes=data/tmp/node.db (temp node db location)
0:00:00 DEB - argument: temp_multipolygons=data/tmp/multipolygon.db (temp multipolygon db location)
0:00:00 DEB - argument: temp_features=data/tmp/feature.db (temp feature db location)
0:00:00 DEB - argument: osm_parse_node_bounds=false (parse bounds from OSM nodes instead of header)
0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit)
0:00:00 DEB - argument: fetch_wikidata=false (fetch wikidata translations then continue)
0:00:00 DEB - argument: use_wikidata=true (use wikidata translations)
0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file)
0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path)
0:00:00 DEB - argument: free_lake_centerlines_after_read=false (delete lake_centerlines input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path)
0:00:00 DEB - argument: free_water_polygons_after_read=false (delete water_polygons input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path)
0:00:00 DEB - argument: free_natural_earth_after_read=false (delete natural_earth input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: osm_path=data/sources/rhode_island.osm.pbf (osm OSM input file path)
0:00:00 DEB - argument: free_osm_after_read=false (delete osm input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: mbtiles=data/out.mbtiles (mbtiles output file)
0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names)
0:00:00 DEB - argument: languages=am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh (languages to use)
0:00:00 DEB - argument: only_layers= (Include only certain layers)
0:00:00 DEB - argument: exclude_layers= (Exclude certain layers)
0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries)
0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13)
0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13)
0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways)
0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield))
0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways)
0:00:00 DEB - argument: transportation_name_minor_refs=false (transportation_name layer: include name and refs from minor road networks if not present on a way)
0:00:00 DEB - argument: mbtiles_name=OpenMapTiles ('name' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a> ('attribution' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_version=3.13.1 ('version' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_type=baselayer ('type' attribute for mbtiles metadata)
0:00:00 DEB - argument: help=false (show arguments then exit)
0:00:00 INF - Building BasemapProfile profile into data/out.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   mbtiles: Encode each tile and write to data/out.mbtiles
0:00:00 INF - no wikidata translations found, run with --fetch-wikidata to download
0:00:00 DEB - ✓ 194M storage on / (/dev/root) requested for read phase disk, 31G available
0:00:00 DEB -  - 43M used for temporary node location cache
0:00:00 DEB -  - 6.6M used for temporary multipolygon geometry cache
0:00:00 DEB -  - 144M used for temporary feature storage
0:00:00 DEB - ✓ 216M storage on / (/dev/root) requested for write phase disk, 31G available
0:00:00 DEB -  - 144M used for temporary feature storage
0:00:00 DEB -  - 72M used for mbtiles output
0:00:00 DEB - ✓ 312M JVM heap requested for read phase, 4.2G available
0:00:00 DEB -  - 300M used for sparsearray node location in-memory index
0:00:00 DEB -  - 12M used for temporary profile storage
0:00:00 DEB - ✓ 50M storage on / (/dev/root) requested for read phase, 31G available
0:00:00 DEB -  - 43M used for sparsearray node location cache
0:00:00 DEB -  - 6.6M used for multipolygon way geometries
0:00:00 DEB - ✓ 50M temporary files and 2.9G of free memory for OS to cache them
0:00:00 INF - Using merge sort feature map, chunk size=1431mb max workers=2
0:00:02 INF - dataFileCache open start
0:00:02 INF [lake_centerlines] - 
0:00:02 INF [lake_centerlines] - Starting...
0:00:04 INF [lake_centerlines] -  read: [  59k 100%  28k/s ] write: [    0    0/s ] 0    
    cpus: 1.9 gc:  4% heap: 159M/4.2G direct: 237k postGC: 74M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:04 INF [lake_centerlines] - Finished in 2s cpu:4s avg:1.9
0:00:04 INF [lake_centerlines] -   read     1x(84% 2s)
0:00:04 INF [lake_centerlines] -   process  2x(13% 0.3s wait:2s)
0:00:04 INF [lake_centerlines] -   write    1x(0% 0s wait:2s)
0:00:04 INF [water_polygons] - 
0:00:04 INF [water_polygons] - Starting...
0:00:14 INF [water_polygons] -  read: [   2k  14%  199/s ] write: [  47k 4.6k/s ] 1.4G 
    cpus: 1.9 gc: 13% heap: 1G/4.2G direct: 52M postGC: 1.1G
    read(57%) ->    (0/1k) -> process(17% 38%) -> (174/53k) -> write( 0%)
0:00:24 INF [water_polygons] -  read: [ 4.7k  33%  272/s ] write: [ 279k  23k/s ] 1.4G 
    cpus: 1.7 gc:  6% heap: 1.8G/4.2G direct: 52M postGC: 1.4G
    read(66%) ->    (0/1k) -> process(37% 24%) -> (514/53k) -> write( 0%)
0:00:31 INF [water_polygons] -  read: [  14k 100% 1.3k/s ] write: [ 4.3M 571k/s ] 186M 
    cpus: 1.6 gc:  7% heap: 2.6G/4.2G direct: 52M postGC: 1.6G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:31 INF [water_polygons] - Finished in 27s cpu:48s gc:2s avg:1.8
0:00:31 INF [water_polygons] -   read     1x(60% 16s wait:3s)
0:00:31 INF [water_polygons] -   process  2x(29% 8s wait:13s)
0:00:31 INF [water_polygons] -   write    1x(2% 0.6s wait:26s)
0:00:31 INF [natural_earth] - unzipping /home/runner/work/planetiler/planetiler/data/sources/natural_earth_vector.sqlite.zip to data/tmp/natearth.sqlite
0:00:38 INF [natural_earth] - 
0:00:38 INF [natural_earth] - Starting...
0:00:48 INF [natural_earth] -  read: [ 349k 100%  36k/s ] write: [  181   18/s ] 186M 
    cpus: 1.6 gc:  0% heap: 1.1G/4.2G direct: 52M postGC: 181M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:48 INF [natural_earth] - Finished in 10s cpu:16s avg:1.5
0:00:48 INF [natural_earth] -   read     1x(86% 9s sys:1s)
0:00:48 INF [natural_earth] -   process  2x(18% 2s wait:9s)
0:00:48 INF [natural_earth] -   write    1x(0% 0s wait:10s)
0:00:48 INF [osm_pass1] - 
0:00:48 INF [osm_pass1] - Starting...
0:00:51 INF [osm_pass1:process] - Finished nodes: 4,599,299 (1.5M/s) in 3s cpu:5s avg:1.7
0:00:52 INF [osm_pass1:process] - Finished ways: 331,643 (359k/s) in 0.9s cpu:2s avg:2
0:00:52 INF [osm_pass1:process] - Finished relations: 7,745 (135k/s) in 0.1s cpu:0.1s avg:2.1
0:00:52 INF [osm_pass1] -  nodes: [ 4.5M 1.1M/s ] 418M  ways: [ 331k  82k/s ] rels: [ 7.7k 1.9k/s ] blocks: [  618  153/s ]
    cpus: 1.8 gc:  0% heap: 1.3G/4.2G direct: 52M postGC: 851M hppc: 879k
    read( -%) ->     (0/4) -> parse( -%) ->     (0/4) -> process( -%)
0:00:52 DEB [osm_pass1] - Processed 618 blocks:
0:00:52 DEB [osm_pass1] -   nodes: 4,599,299 (1.5M/s) in 3s cpu:5s avg:1.7
0:00:52 DEB [osm_pass1] -   ways: 331,643 (359k/s) in 0.9s cpu:2s avg:2
0:00:52 DEB [osm_pass1] -   relations: 7,745 (135k/s) in 0.1s cpu:0.1s avg:2.1
0:00:52 INF [osm_pass1] - Finished in 4s cpu:7s avg:1.8
0:00:52 INF [osm_pass1] -   read     1x(2% 0.1s wait:4s)
0:00:52 INF [osm_pass1] -   parse    1x(64% 3s)
0:00:52 INF [osm_pass1] -   process  1x(48% 2s wait:2s)
0:00:52 INF [osm_pass2] - 
0:00:52 INF [osm_pass2] - Starting...
0:00:54 DEB [osm_pass2:process] - Sorting long long multimap...
0:00:54 INF [osm_pass2:process] - Finished nodes: 4,599,299 (2M/s) in 2s cpu:5s avg:2
0:00:55 DEB [osm_pass2:process] - Sorted long long multimap 0s cpu:0s avg:1.8
0:00:55 WAR [osm_pass2:process] - No GB polygon for inferring route network types
0:01:02 INF [osm_pass2] -  nodes: [ 4.5M 100% 459k/s ] 418M  ways: [ 163k  49%  16k/s ] rels: [    0   0%    0/s ] features: [ 4.9M  59k/s ] 1.6G  blocks: [  594  96%   59/s ]
    cpus: 2 gc:  0% heap: 2.6G/4.2G direct: 53M postGC: 866M relInfo: 827k mpGeoms: 649k 
    read( 0%) ->   (11/13) -> process(64% 61%) -> (1.7k/53k) -> write( 2%)
0:01:11 INF [osm_pass2:process] - Finished ways: 331,643 (19k/s) in 17s cpu:33s avg:2
0:01:12 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 418M  ways: [ 331k 100%  16k/s ] rels: [ 1.2k  17%  129/s ] features: [ 5.2M  34k/s ] 1.6G  blocks: [  617 100%    2/s ]
    cpus: 2 gc:  1% heap: 2.7G/4.2G direct: 52M postGC: 868M relInfo: 827k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process(81% 82%) -> (1.3k/53k) -> write( 1%)
0:01:21 INF [osm_pass2:process] - Finished relations: 7,745 (829/s) in 9s cpu:19s avg:2
0:01:22 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 418M  ways: [ 331k 100%    0/s ] rels: [ 7.1k  93%  589/s ] features: [ 5.2M 2.7k/s ] 1.6G  blocks: [  618 100%   <1/s ]
    cpus: 2 gc:  1% heap: 2G/4.2G direct: 52M postGC: 860M relInfo: 827k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process(84% 84%) -> (1.4k/53k) -> write( 0%)
0:01:23 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 418M  ways: [ 331k 100%    0/s ] rels: [ 7.7k 100%  461/s ] features: [ 5.2M 3.5k/s ] 270M  blocks: [  618 100%    0/s ]
    cpus: 2 gc:  1% heap: 900M/4.2G direct: 52M postGC: 860M relInfo: 827k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process( -%  -%) ->   (0/53k) -> write( 1%)
0:01:23 DEB [osm_pass2] - Processed 618 blocks:
0:01:23 DEB [osm_pass2] -   nodes: 4,599,299 (2M/s) in 2s cpu:5s avg:2
0:01:23 DEB [osm_pass2] -   ways: 331,643 (19k/s) in 17s cpu:33s avg:2
0:01:23 DEB [osm_pass2] -   relations: 7,745 (829/s) in 9s cpu:19s avg:2
0:01:23 INF [osm_pass2] - Finished in 31s cpu:1m2s avg:2
0:01:23 INF [osm_pass2] -   read     1x(0% 0s wait:15s done:17s)
0:01:23 INF [osm_pass2] -   process  2x(76% 24s)
0:01:23 INF [osm_pass2] -   write    1x(1% 0.4s wait:31s)
0:01:23 INF [boundaries] - 
0:01:23 INF [boundaries] - Starting...
0:01:23 INF [boundaries] - Creating polygons for 1 boundaries
0:01:23 WAR [boundaries] - Unable to form closed polygon for OSM relation 148838 (likely missing edges)
0:01:23 INF [boundaries] - Finished creating 0 country polygons
0:01:24 INF [boundaries] - Finished in 0s cpu:0.1s avg:1.8
0:01:24 INF - Deleting node.db to make room for output file
0:01:24 INF [sort] - 
0:01:24 INF [sort] - Starting...
0:01:24 INF [sort] - Grouped 8 chunks into 1
0:01:26 INF [sort] -  chunks: [   1 /   1 100% ] 270M 
    cpus: 1.2 gc:  0% heap: 1.3G/4.2G direct: 52M postGC: 860M
    ->     (0/3) -> worker( -%)
0:01:26 INF [sort] - Finished in 3s cpu:3s avg:1.2
0:01:27 INF [sort] -   worker  1x(75% 2s)
0:01:27 INF - read:1s write:0s sort:1s
0:01:27 INF [mbtiles] - 
0:01:27 INF [mbtiles] - Starting...
0:01:27 DEB [mbtiles:write] - Execute mbtiles: create table metadata (name text, value text);
0:01:27 DEB [mbtiles:write] - Execute mbtiles: create unique index name on metadata (name);
0:01:27 DEB [mbtiles:write] - Execute mbtiles: create table tiles_shallow (
  zoom_level integer,
  tile_column integer,
  tile_row integer,
  tile_data_id integer

  , primary key(zoom_level,tile_column,tile_row)

) without rowid

0:01:27 DEB [mbtiles:write] - Execute mbtiles: create table tiles_data (
  tile_data_id integer primary key,
  tile_data blob
)

0:01:27 DEB [mbtiles:write] - Execute mbtiles: create view tiles AS
select
  tiles_shallow.zoom_level as zoom_level,
  tiles_shallow.tile_column as tile_column,
  tiles_shallow.tile_row as tile_row,
  tiles_data.tile_data as tile_data
from tiles_shallow
join tiles_data on tiles_shallow.tile_data_id = tiles_data.tile_data_id

0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: name=OpenMapTiles
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: format=pbf
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: version=3.13.1
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: type=baselayer
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: bounds=-74.07,21.34,-17.84,43.55
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: center=-45.955,32.445,3
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: minzoom=0
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: maxzoom=14
0:01:27 DEB [mbtiles:write] - Set mbtiles metadata: json={"vector_layers":[{"id":"aerodrome_label","fields":{"name_int":"String","iata":"String","ele_ft":"Number","name_de":"String","name":"String","icao":"String","name:en":"String","class":"String","ele":"Number","name_en":"String","name:latin":"String"},"minzoom":10,"maxzoom":14},{"id":"aeroway","fields":{"ref":"String","class":"String"},"minzoom":10,"maxzoom":14},{"id":"boundary","fields":{"disputed":"Number","admin_level":"Number","maritime":"Number"},"minzoom":0,"maxzoom":14},{"id":"building","fields":{"colour":"String","render_height":"Number","render_min_height":"Number"},"minzoom":13,"maxzoom":14},{"id":"housenumber","fields":{"housenumber":"String"},"minzoom":14,"maxzoom":14},{"id":"landcover","fields":{"subclass":"String","class":"String","_numpoints":"Number"},"minzoom":7,"maxzoom":14},{"id":"landuse","fields":{"class":"String"},"minzoom":4,"maxzoom":14},{"id":"mountain_peak","fields":{"name_int":"String","customary_ft":"Number","ele_ft":"Number","name_de":"String","name":"String","rank":"Number","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":7,"maxzoom":14},{"id":"park","fields":{"name_int":"String","name_de":"String","name":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"place","fields":{"name:fy":"String","name_int":"String","capital":"Number","name:uk":"String","name:pl":"String","name:nl":"String","name:be":"String","name:ru":"String","name:ko":"String","name_de":"String","name":"String","rank":"Number","name:en":"String","name:eo":"String","class":"String","name:hu":"String","name:ta":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":2,"maxzoom":14},{"id":"poi","fields":{"name_int":"String","level":"Number","name_de":"String","name":"String","subclass":"String","indoor":"Number","name:en":"String","class":"String","layer":"Number","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":12,"maxzoom":14},{"id":"transportation","fields":{"access":"String","brunnel":"String","expressway":"Number","surface":"String","bicycle":"String","level":"Number","ramp":"Number","mtb_scale":"String","toll":"Number","oneway":"Number","layer":"Number","network":"String","horse":"String","service":"String","subclass":"String","class":"String","foot":"String"},"minzoom":4,"maxzoom":14},{"id":"transportation_name","fields":{"name_int":"String","route_4":"String","route_3":"String","route_2":"String","route_1":"String","layer":"Number","network":"String","ref":"String","name_de":"String","name":"String","subclass":"String","ref_length":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"water","fields":{"intermittent":"Number","id":"Number","class":"String"},"minzoom":0,"maxzoom":14},{"id":"water_name","fields":{"name_int":"String","name_de":"String","name":"String","intermittent":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":9,"maxzoom":14},{"id":"waterway","fields":{"name_int":"String","brunnel":"String","name_de":"String","_relid":"Number","intermittent":"Number","name":"String","class":"String","name:latin":"String","name_en":"String"},"minzoom":4,"maxzoom":14}]}
0:01:29 INF [mbtiles:write] - Starting z0
0:01:29 INF [mbtiles:write] - Finished z0 in 0s cpu:0s avg:0, now starting z1
0:01:29 INF [mbtiles:write] - Finished z1 in 0s cpu:0s avg:0, now starting z2
0:01:29 INF [mbtiles:write] - Finished z2 in 0s cpu:0s avg:0, now starting z3
0:01:29 INF [mbtiles:write] - Finished z3 in 0s cpu:0s avg:0, now starting z4
0:01:29 INF [mbtiles:write] - Finished z4 in 0s cpu:0s avg:0, now starting z5
0:01:29 INF [mbtiles:write] - Finished z5 in 0s cpu:0s avg:0, now starting z6
0:01:29 INF [mbtiles:write] - Finished z6 in 0s cpu:0s avg:0, now starting z7
0:01:29 INF [mbtiles:write] - Finished z7 in 0.5s cpu:1s avg:2, now starting z8
0:01:31 INF [mbtiles:write] - Finished z8 in 2s cpu:4s avg:2, now starting z9
0:01:32 INF [mbtiles:write] - Finished z9 in 1s cpu:2s avg:2, now starting z10
0:01:33 INF [mbtiles:write] - Finished z10 in 1s cpu:2s avg:2, now starting z11
0:01:35 INF [mbtiles:write] - Finished z11 in 2s cpu:3s avg:2, now starting z12
0:01:37 INF [mbtiles] -  features: [ 364k   7%  36k/s ] 270M  tiles: [ 148k  14k/s ] 10M  
    cpus: 2 gc:  3% heap: 1.9G/4.2G direct: 52M postGC: 803M
    read( 1%) -> (214/217) -> encode(58% 64%) -> (215/216) -> write( 4%)
    last tile: 12/1486/1781 (z12 43%) https://www.openstreetmap.org/#map=12/22.83695/-49.39453
0:01:38 INF [mbtiles:write] - Finished z12 in 3s cpu:6s avg:2, now starting z13
0:01:47 INF [mbtiles] -  features: [ 781k  15%  41k/s ] 270M  tiles: [ 291k  14k/s ] 13M  
    cpus: 2 gc:  4% heap: 2.2G/4.2G direct: 52M postGC: 903M
    read( 0%) -> (214/217) -> encode(64% 61%) -> (215/216) -> write( 3%)
    last tile: 13/2471/3047 (z13 4%) https://www.openstreetmap.org/#map=13/41.80408/-71.41113
0:01:51 INF [mbtiles:write] - Finished z13 in 13s cpu:26s avg:2, now starting z14
0:01:57 INF [mbtiles] -  features: [ 4.2M  80% 345k/s ] 270M  tiles: [ 2.8M 257k/s ] 86M  
    cpus: 1.9 gc:  2% heap: 2.2G/4.2G direct: 52M postGC: 874M
    read( 4%) ->   (0/217) -> encode(52% 54%) -> (215/216) -> write(49%)
    last tile: 14/6358/6159 (z14 60%) https://www.openstreetmap.org/#map=14/40.73061/-40.29785
0:01:59 DEB [mbtiles:write] - Shallow tiles written: 4,115,450
0:01:59 DEB [mbtiles:write] - Tile data written: 21,295 (99% omitted)
0:01:59 DEB [mbtiles:write] - Unique tile hashes: 11,269
0:01:59 INF [mbtiles:write] - Finished z14 in 8s cpu:15s avg:1.8
0:01:59 INF [mbtiles] -  features: [ 5.2M 100% 415k/s ] 270M  tiles: [ 4.1M 487k/s ] 108M 
    cpus: 1.6 gc:  3% heap: 979M/4.2G direct: 52M postGC: 875M
    read( -%) ->   (0/217) -> encode( -%  -%) ->   (0/216) -> write( -%)
    last tile: 14/7380/5985 (z14 100%) https://www.openstreetmap.org/#map=14/43.56447/-17.84180
0:01:59 DEB [mbtiles] - Tile stats:
0:01:59 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:01:59 DEB [mbtiles] - z1 avg:4k max:4k
0:01:59 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:01:59 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:01:59 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:01:59 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:01:59 DEB [mbtiles] - z6 avg:973 max:22k
0:01:59 DEB [mbtiles] - z7 avg:751 max:55k
0:01:59 DEB [mbtiles] - z8 avg:394 max:112k
0:01:59 DEB [mbtiles] - z9 avg:270 max:278k
0:01:59 DEB [mbtiles] - z10 avg:154 max:233k
0:01:59 DEB [mbtiles] - z11 avg:102 max:132k
0:01:59 DEB [mbtiles] - z12 avg:83 max:119k
0:01:59 DEB [mbtiles] - z13 avg:71 max:109k
0:01:59 DEB [mbtiles] - z14 avg:67 max:266k
0:01:59 DEB [mbtiles] - all avg:69 max:0
0:01:59 DEB [mbtiles] -  # features: 5,293,577
0:01:59 DEB [mbtiles] -     # tiles: 4,115,450
0:01:59 INF [mbtiles] - Finished in 33s cpu:1m3s avg:1.9
0:01:59 INF [mbtiles] -   read    1x(2% 0.7s wait:31s)
0:01:59 INF [mbtiles] -   encode  2x(56% 18s wait:3s)
0:01:59 INF [mbtiles] -   write   1x(25% 8s wait:23s)
0:01:59 INF - Finished in 2m cpu:3m35s gc:4s avg:1.8
0:01:59 INF - FINISHED!
0:01:59 INF - 
0:01:59 INF - ----------------------------------------
0:01:59 INF - 	overall          2m cpu:3m35s gc:4s avg:1.8
0:01:59 INF - 	lake_centerlines 2s cpu:4s avg:1.9
0:01:59 INF - 	  read     1x(84% 2s)
0:01:59 INF - 	  process  2x(13% 0.3s wait:2s)
0:01:59 INF - 	  write    1x(0% 0s wait:2s)
0:01:59 INF - 	water_polygons   27s cpu:48s gc:2s avg:1.8
0:01:59 INF - 	  read     1x(60% 16s wait:3s)
0:01:59 INF - 	  process  2x(29% 8s wait:13s)
0:01:59 INF - 	  write    1x(2% 0.6s wait:26s)
0:01:59 INF - 	natural_earth    10s cpu:16s avg:1.5
0:01:59 INF - 	  read     1x(86% 9s sys:1s)
0:01:59 INF - 	  process  2x(18% 2s wait:9s)
0:01:59 INF - 	  write    1x(0% 0s wait:10s)
0:01:59 INF - 	osm_pass1        4s cpu:7s avg:1.8
0:01:59 INF - 	  read     1x(2% 0.1s wait:4s)
0:01:59 INF - 	  parse    1x(64% 3s)
0:01:59 INF - 	  process  1x(48% 2s wait:2s)
0:01:59 INF - 	osm_pass2        31s cpu:1m2s avg:2
0:01:59 INF - 	  read     1x(0% 0s wait:15s done:17s)
0:01:59 INF - 	  process  2x(76% 24s)
0:01:59 INF - 	  write    1x(1% 0.4s wait:31s)
0:01:59 INF - 	boundaries       0s cpu:0.1s avg:1.8
0:01:59 INF - 	sort             3s cpu:3s avg:1.2
0:01:59 INF - 	  worker  1x(75% 2s)
0:01:59 INF - 	mbtiles          33s cpu:1m3s avg:1.9
0:01:59 INF - 	  read    1x(2% 0.7s wait:31s)
0:01:59 INF - 	  encode  2x(56% 18s wait:3s)
0:01:59 INF - 	  write   1x(25% 8s wait:23s)
0:01:59 INF - ----------------------------------------
0:01:59 INF - 	features	270MB
0:01:59 INF - 	mbtiles	108MB
-rw-r--r-- 1 runner docker 55M Jul 11 07:00 run.jar

ℹ️ This Branch Logs 5ab10cb

0:00:00 DEB - argument: config=null (path to config file)
0:00:00 DEB - argument: area=rhode island (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet'))
0:00:00 INF - Using in-memory stats
0:00:00 INF [overall] - 
0:00:00 INF [overall] - Starting...
0:00:00 DEB - argument: madvise=true (default value for whether to use linux madvise(random) to improve memory-mapped read performance for temporary storage)
0:00:00 DEB - argument: storage=mmap (default storage type for temporary data, one of [ram, mmap, direct])
0:00:00 DEB - argument: threads=2 (num threads)
0:00:00 DEB - argument: write_threads=1 (number of threads to use when writing temp features)
0:00:00 DEB - argument: process_threads=2 (number of threads to use when processing input features)
0:00:00 DEB - argument: bounds=Env[-74.07 : -17.84, 21.34 : 43.55] (bounds)
0:00:00 DEB - argument: feature_read_threads=1 (number of threads to use when reading features at tile write time)
0:00:00 DEB - argument: loginterval=10 seconds (time between logs)
0:00:00 DEB - argument: minzoom=0 (minimum zoom level)
0:00:00 DEB - argument: maxzoom=14 (maximum zoom level (limit 14))
0:00:00 DEB - argument: render_maxzoom=14 (maximum rendering zoom level (limit 14))
0:00:00 DEB - argument: skip_mbtiles_index_creation=false (skip adding index to mbtiles file)
0:00:00 DEB - argument: optimize_db=false (optimize mbtiles after writing)
0:00:00 DEB - argument: emit_tiles_in_order=true (emit tiles in index order)
0:00:00 DEB - argument: force=false (overwriting output file and ignore disk/RAM warnings)
0:00:00 DEB - argument: gzip_temp=false (gzip temporary feature storage (uses more CPU, but less disk space))
0:00:00 DEB - argument: mmap_temp=true (use memory-mapped IO for temp feature files)
0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks)
0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks)
0:00:00 DEB - argument: nodemap_type=sparsearray (type of node location map, one of [noop, sortedtable, sparsearray, array])
0:00:00 DEB - argument: nodemap_storage=mmap (storage for node location map, one of [ram, mmap, direct])
0:00:00 DEB - argument: nodemap_madvise=true (use linux madvise(random) for node locations)
0:00:00 DEB - argument: multipolygon_geometry_storage=mmap (storage for multipolygon geometries, one of [ram, mmap, direct])
0:00:00 DEB - argument: multipolygon_geometry_madvise=true (use linux madvise(random) for temporary multipolygon geometry storage)
0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP)
0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP)
0:00:00 DEB - argument: http_retries=1 (Retries to use when downloading files over HTTP)
0:00:00 DEB - argument: download_chunk_size_mb=100 (Size of file chunks to download in parallel in megabytes)
0:00:00 DEB - argument: download_threads=1 (Number of parallel threads to use when downloading each file)
0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level)
0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level)
0:00:00 DEB - argument: osm_lazy_reads=false (Read OSM blocks from disk in worker threads)
0:00:00 DEB - argument: compact_db=true (Reduce the DB size by separating and deduping the tile data)
0:00:00 DEB - argument: skip_filled_tiles=false (Skip writing tiles containing only polygon fills to the output)
0:00:00 DEB - argument: tmpdir=data/tmp (temp directory)
0:00:00 DEB - argument: only_download=false (download source data then exit)
0:00:00 DEB - argument: download=false (download sources)
0:00:00 DEB - argument: temp_nodes=data/tmp/node.db (temp node db location)
0:00:00 DEB - argument: temp_multipolygons=data/tmp/multipolygon.db (temp multipolygon db location)
0:00:00 DEB - argument: temp_features=data/tmp/feature.db (temp feature db location)
0:00:00 DEB - argument: osm_parse_node_bounds=false (parse bounds from OSM nodes instead of header)
0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit)
0:00:00 DEB - argument: fetch_wikidata=false (fetch wikidata translations then continue)
0:00:00 DEB - argument: use_wikidata=true (use wikidata translations)
0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file)
0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path)
0:00:00 DEB - argument: free_lake_centerlines_after_read=false (delete lake_centerlines input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path)
0:00:00 DEB - argument: free_water_polygons_after_read=false (delete water_polygons input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path)
0:00:00 DEB - argument: free_natural_earth_after_read=false (delete natural_earth input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: osm_path=data/sources/rhode_island.osm.pbf (osm OSM input file path)
0:00:00 DEB - argument: free_osm_after_read=false (delete osm input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: mbtiles=data/out.mbtiles (mbtiles output file)
0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names)
0:00:00 DEB - argument: languages=am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh (languages to use)
0:00:00 DEB - argument: only_layers= (Include only certain layers)
0:00:00 DEB - argument: exclude_layers= (Exclude certain layers)
0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries)
0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13)
0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13)
0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways)
0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield))
0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways)
0:00:00 DEB - argument: transportation_name_minor_refs=false (transportation_name layer: include name and refs from minor road networks if not present on a way)
0:00:00 DEB - argument: mbtiles_name=OpenMapTiles ('name' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a> ('attribution' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_version=3.13.1 ('version' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_type=baselayer ('type' attribute for mbtiles metadata)
0:00:00 DEB - argument: help=false (show arguments then exit)
0:00:00 INF - Building BasemapProfile profile into data/out.mbtiles in these phases:
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   mbtiles: Encode each tile and write to data/out.mbtiles
0:00:00 INF - no wikidata translations found, run with --fetch-wikidata to download
0:00:00 DEB - ✓ 194M storage on / (/dev/root) requested for read phase disk, 31G available
0:00:00 DEB -  - 43M used for temporary node location cache
0:00:00 DEB -  - 6.6M used for temporary multipolygon geometry cache
0:00:00 DEB -  - 144M used for temporary feature storage
0:00:00 DEB - ✓ 216M storage on / (/dev/root) requested for write phase disk, 31G available
0:00:00 DEB -  - 144M used for temporary feature storage
0:00:00 DEB -  - 72M used for mbtiles output
0:00:00 DEB - ✓ 312M JVM heap requested for read phase, 4.2G available
0:00:00 DEB -  - 300M used for sparsearray node location in-memory index
0:00:00 DEB -  - 12M used for temporary profile storage
0:00:00 DEB - ✓ 50M storage on / (/dev/root) requested for read phase, 31G available
0:00:00 DEB -  - 43M used for sparsearray node location cache
0:00:00 DEB -  - 6.6M used for multipolygon way geometries
0:00:00 DEB - ✓ 50M temporary files and 2.9G of free memory for OS to cache them
0:00:00 INF - Using merge sort feature map, chunk size=1431mb max workers=2
0:00:02 INF - dataFileCache open start
0:00:02 INF [lake_centerlines] - 
0:00:02 INF [lake_centerlines] - Starting...
0:00:04 INF [lake_centerlines] -  read: [  59k 100%  27k/s ] write: [    0    0/s ] 0    
    cpus: 1.7 gc:  4% heap: 159M/4.2G direct: 237k postGC: 74M
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:04 INF [lake_centerlines] - Finished in 2s cpu:4s avg:1.8
0:00:04 INF [lake_centerlines] -   read     1x(77% 2s)
0:00:04 INF [lake_centerlines] -   process  2x(11% 0.3s wait:2s)
0:00:04 INF [lake_centerlines] -   write    1x(0% 0s wait:2s)
0:00:04 INF [water_polygons] - 
0:00:04 INF [water_polygons] - Starting...
0:00:14 INF [water_polygons] -  read: [ 1.9k  13%  194/s ] write: [  46k 4.5k/s ] 1.4G 
    cpus: 1.9 gc: 12% heap: 1.2G/4.2G direct: 52M postGC: 1G
    read(58%) ->    (0/1k) -> process(17% 38%) -> (1.1k/53k) -> write( 0%)
0:00:24 INF [water_polygons] -  read: [ 4.6k  32%  265/s ] write: [ 233k  18k/s ] 1.4G 
    cpus: 1.7 gc:  9% heap: 2.3G/4.2G direct: 52M postGC: 1.4G
    read(62%) ->    (1/1k) -> process(26% 30%) ->  (1k/53k) -> write( 0%)
0:00:32 INF [water_polygons] -  read: [  14k 100% 1.2k/s ] write: [ 4.3M 511k/s ] 186M 
    cpus: 1.5 gc:  7% heap: 2.6G/4.2G direct: 52M postGC: 1.7G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:32 INF [water_polygons] - Finished in 28s cpu:48s gc:3s avg:1.7
0:00:32 INF [water_polygons] -   read     1x(58% 16s wait:3s)
0:00:32 INF [water_polygons] -   process  2x(28% 8s wait:13s)
0:00:32 INF [water_polygons] -   write    1x(2% 0.6s wait:27s)
0:00:32 INF [natural_earth] - unzipping /home/runner/work/planetiler/planetiler/data/sources/natural_earth_vector.sqlite.zip to data/tmp/natearth.sqlite
0:00:41 INF [natural_earth] - 
0:00:41 INF [natural_earth] - Starting...
0:00:52 INF [natural_earth] -  read: [ 349k 100%  36k/s ] write: [  181   18/s ] 186M 
    cpus: 1.6 gc:  1% heap: 1.9G/4.2G direct: 52M postGC: 1.7G
    read( -%) ->    (0/1k) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:00:52 INF [natural_earth] - Finished in 10s cpu:16s avg:1.6
0:00:52 INF [natural_earth] -   read     1x(87% 9s sys:1s)
0:00:52 INF [natural_earth] -   process  2x(20% 2s wait:9s)
0:00:52 INF [natural_earth] -   write    1x(0% 0s wait:10s)
0:00:52 INF [osm_pass1] - 
0:00:52 INF [osm_pass1] - Starting...
0:00:55 INF [osm_pass1:process] - Finished nodes: 4,599,299 (1.4M/s) in 3s cpu:5s avg:1.4
0:00:57 INF [osm_pass1:process] - Finished ways: 331,643 (268k/s) in 1s cpu:2s avg:1.4
0:00:57 INF [osm_pass1:process] - Finished relations: 7,745 (107k/s) in 0.1s cpu:0.1s avg:2.1
0:00:57 INF [osm_pass1] -  nodes: [ 4.5M 993k/s ] 418M  ways: [ 331k  71k/s ] rels: [ 7.7k 1.6k/s ] blocks: [  618  133/s ]
    cpus: 1.5 gc:  1% heap: 1.9G/4.2G direct: 52M postGC: 2.2G hppc: 879k
    read( -%) ->     (0/4) -> parse( -%) ->     (0/4) -> process( -%)
0:00:57 DEB [osm_pass1] - Processed 618 blocks:
0:00:57 DEB [osm_pass1] -   nodes: 4,599,299 (1.4M/s) in 3s cpu:5s avg:1.4
0:00:57 DEB [osm_pass1] -   ways: 331,643 (268k/s) in 1s cpu:2s avg:1.4
0:00:57 DEB [osm_pass1] -   relations: 7,745 (107k/s) in 0.1s cpu:0.1s avg:2.1
0:00:57 INF [osm_pass1] - Finished in 5s cpu:7s avg:1.5
0:00:57 INF [osm_pass1] -   read     1x(3% 0.1s wait:4s)
0:00:57 INF [osm_pass1] -   parse    1x(53% 2s wait:1s)
0:00:57 INF [osm_pass1] -   process  1x(35% 2s wait:2s)
0:00:57 INF [osm_pass2] - 
0:00:57 INF [osm_pass2] - Starting...
0:01:00 DEB [osm_pass2:process] - Sorting long long multimap...
0:01:00 INF [osm_pass2:process] - Finished nodes: 4,599,299 (1.5M/s) in 3s cpu:6s avg:2
0:01:00 DEB [osm_pass2:process] - Sorted long long multimap 0s cpu:0s avg:1.7
0:01:00 WAR [osm_pass2:process] - No GB polygon for inferring route network types
0:01:07 INF [osm_pass2] -  nodes: [ 4.5M 100% 459k/s ] 418M  ways: [ 130k  39%  13k/s ] rels: [    0   0%    0/s ] features: [ 4.8M  52k/s ] 1.6G  blocks: [  591  96%   59/s ]
    cpus: 2 gc:  1% heap: 3.2G/4.2G direct: 53M postGC: 852M relInfo: 827k mpGeoms: 624k 
    read( 0%) ->   (11/13) -> process(64% 65%) -> (1.5k/53k) -> write( 2%)
0:01:17 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 418M  ways: [ 316k  96%  18k/s ] rels: [    0   0%    0/s ] features: [ 5.2M  37k/s ] 1.6G  blocks: [  614  99%    2/s ]
    cpus: 2 gc:  1% heap: 3.1G/4.2G direct: 52M postGC: 863M relInfo: 827k mpGeoms: 19M  
    read( -%) ->    (2/13) -> process(84% 82%) -> (1.7k/53k) -> write( 1%)
0:01:17 INF [osm_pass2:process] - Finished ways: 331,643 (18k/s) in 18s cpu:35s avg:2
0:01:26 INF [osm_pass2:process] - Finished relations: 7,745 (855/s) in 9s cpu:18s avg:2
0:01:27 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 418M  ways: [ 331k 100% 1.4k/s ] rels: [ 6.8k  88%  684/s ] features: [ 5.2M 6.6k/s ] 1.6G  blocks: [  618 100%   <1/s ]
    cpus: 2 gc:  1% heap: 2.1G/4.2G direct: 52M postGC: 854M relInfo: 827k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process(83% 82%) -> (1.4k/53k) -> write( 0%)
0:01:29 INF [osm_pass2] -  nodes: [ 4.5M 100%    0/s ] 418M  ways: [ 331k 100%    0/s ] rels: [ 7.7k 100%  380/s ] features: [ 5.2M 3.1k/s ] 270M  blocks: [  618 100%    0/s ]
    cpus: 2 gc:  0% heap: 2.5G/4.2G direct: 52M postGC: 846M relInfo: 827k mpGeoms: 19M  
    read( -%) ->    (0/13) -> process( -%  -%) ->   (0/53k) -> write( -%)
0:01:29 DEB [osm_pass2] - Processed 618 blocks:
0:01:29 DEB [osm_pass2] -   nodes: 4,599,299 (1.5M/s) in 3s cpu:6s avg:2
0:01:29 DEB [osm_pass2] -   ways: 331,643 (18k/s) in 18s cpu:35s avg:2
0:01:29 DEB [osm_pass2] -   relations: 7,745 (855/s) in 9s cpu:18s avg:2
0:01:29 INF [osm_pass2] - Finished in 32s cpu:1m4s avg:2
0:01:29 INF [osm_pass2] -   read     1x(0% 0.1s wait:16s done:17s)
0:01:29 INF [osm_pass2] -   process  2x(77% 25s)
0:01:29 INF [osm_pass2] -   write    1x(1% 0.4s wait:32s)
0:01:29 INF [boundaries] - 
0:01:29 INF [boundaries] - Starting...
0:01:29 INF [boundaries] - Creating polygons for 1 boundaries
0:01:29 WAR [boundaries] - Unable to form closed polygon for OSM relation 148838 (likely missing edges)
0:01:29 INF [boundaries] - Finished creating 0 country polygons
0:01:29 INF [boundaries] - Finished in 0s cpu:0.1s avg:1.6
0:01:29 INF - Deleting node.db to make room for output file
0:01:29 INF [sort] - 
0:01:29 INF [sort] - Starting...
0:01:29 INF [sort] - Grouped 8 chunks into 1
0:01:32 INF [sort] -  chunks: [   1 /   1 100% ] 270M 
    cpus: 1.2 gc:  0% heap: 3G/4.2G direct: 52M postGC: 846M
    ->     (0/3) -> worker( -%)
0:01:32 INF [sort] - Finished in 3s cpu:3s avg:1.2
0:01:32 INF [sort] -   worker  1x(74% 2s)
0:01:32 INF - read:0s write:0s sort:1s
0:01:32 INF [mbtiles] - 
0:01:32 INF [mbtiles] - Starting...
0:01:33 DEB [mbtiles:write] - Execute mbtiles: create table metadata (name text, value text);
0:01:33 DEB [mbtiles:write] - Execute mbtiles: create unique index name on metadata (name);
0:01:33 DEB [mbtiles:write] - Execute mbtiles: create table tiles_shallow (
  zoom_level integer,
  tile_column integer,
  tile_row integer,
  tile_data_id integer

  , primary key(zoom_level,tile_column,tile_row)

) without rowid

0:01:33 DEB [mbtiles:write] - Execute mbtiles: create table tiles_data (
  tile_data_id integer primary key,
  tile_data blob
)

0:01:33 DEB [mbtiles:write] - Execute mbtiles: create view tiles AS
select
  tiles_shallow.zoom_level as zoom_level,
  tiles_shallow.tile_column as tile_column,
  tiles_shallow.tile_row as tile_row,
  tiles_data.tile_data as tile_data
from tiles_shallow
join tiles_data on tiles_shallow.tile_data_id = tiles_data.tile_data_id

0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: name=OpenMapTiles
0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: format=pbf
0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org
0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a>
0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: version=3.13.1
0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: type=baselayer
0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: bounds=-74.07,21.34,-17.84,43.55
0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: center=-45.955,32.445,3
0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: minzoom=0
0:01:33 DEB [mbtiles:write] - Set mbtiles metadata: maxzoom=14
0:01:34 DEB [mbtiles:write] - Set mbtiles metadata: json={"vector_layers":[{"id":"aerodrome_label","fields":{"name_int":"String","iata":"String","ele_ft":"Number","name_de":"String","name":"String","icao":"String","name:en":"String","class":"String","ele":"Number","name_en":"String","name:latin":"String"},"minzoom":10,"maxzoom":14},{"id":"aeroway","fields":{"ref":"String","class":"String"},"minzoom":10,"maxzoom":14},{"id":"boundary","fields":{"disputed":"Number","admin_level":"Number","maritime":"Number"},"minzoom":0,"maxzoom":14},{"id":"building","fields":{"colour":"String","render_height":"Number","render_min_height":"Number"},"minzoom":13,"maxzoom":14},{"id":"housenumber","fields":{"housenumber":"String"},"minzoom":14,"maxzoom":14},{"id":"landcover","fields":{"subclass":"String","class":"String","_numpoints":"Number"},"minzoom":7,"maxzoom":14},{"id":"landuse","fields":{"class":"String"},"minzoom":4,"maxzoom":14},{"id":"mountain_peak","fields":{"name_int":"String","customary_ft":"Number","ele_ft":"Number","name_de":"String","name":"String","rank":"Number","class":"String","name_en":"String","name:latin":"String","ele":"Number"},"minzoom":7,"maxzoom":14},{"id":"park","fields":{"name_int":"String","name_de":"String","name":"String","name:en":"String","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"place","fields":{"name:fy":"String","name_int":"String","capital":"Number","name:uk":"String","name:pl":"String","name:nl":"String","name:be":"String","name:ru":"String","name:ko":"String","name_de":"String","name":"String","rank":"Number","name:en":"String","name:eo":"String","class":"String","name:hu":"String","name:ta":"String","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":2,"maxzoom":14},{"id":"poi","fields":{"name_int":"String","level":"Number","name_de":"String","name":"String","subclass":"String","indoor":"Number","name:en":"String","class":"String","layer":"Number","name:zh":"String","name_en":"String","name:latin":"String"},"minzoom":12,"maxzoom":14},{"id":"transportation","fields":{"access":"String","brunnel":"String","expressway":"Number","surface":"String","bicycle":"String","level":"Number","ramp":"Number","mtb_scale":"String","toll":"Number","oneway":"Number","layer":"Number","network":"String","horse":"String","service":"String","subclass":"String","class":"String","foot":"String"},"minzoom":4,"maxzoom":14},{"id":"transportation_name","fields":{"name_int":"String","route_4":"String","route_3":"String","route_2":"String","route_1":"String","layer":"Number","network":"String","ref":"String","name_de":"String","name":"String","subclass":"String","ref_length":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":6,"maxzoom":14},{"id":"water","fields":{"intermittent":"Number","id":"Number","class":"String"},"minzoom":0,"maxzoom":14},{"id":"water_name","fields":{"name_int":"String","name_de":"String","name":"String","intermittent":"Number","class":"String","name_en":"String","name:latin":"String"},"minzoom":9,"maxzoom":14},{"id":"waterway","fields":{"name_int":"String","brunnel":"String","name_de":"String","_relid":"Number","intermittent":"Number","name":"String","class":"String","name:latin":"String","name_en":"String"},"minzoom":4,"maxzoom":14}]}
0:01:35 INF [mbtiles:write] - Starting z0
0:01:35 INF [mbtiles:write] - Finished z0 in 0s cpu:0s avg:11.9, now starting z1
0:01:35 INF [mbtiles:write] - Finished z1 in 0s cpu:0s avg:0, now starting z2
0:01:35 INF [mbtiles:write] - Finished z2 in 0s cpu:0s avg:0, now starting z3
0:01:35 INF [mbtiles:write] - Finished z3 in 0s cpu:0s avg:0, now starting z4
0:01:35 INF [mbtiles:write] - Finished z4 in 0s cpu:0s avg:0, now starting z5
0:01:35 INF [mbtiles:write] - Finished z5 in 0s cpu:0s avg:9.7, now starting z6
0:01:35 INF [mbtiles:write] - Finished z6 in 0s cpu:0s avg:0, now starting z7
0:01:36 INF [mbtiles:write] - Finished z7 in 0.6s cpu:1s avg:2, now starting z8
0:01:38 INF [mbtiles:write] - Finished z8 in 2s cpu:4s avg:2, now starting z9
0:01:39 INF [mbtiles:write] - Finished z9 in 0.7s cpu:1s avg:2, now starting z10
0:01:40 INF [mbtiles:write] - Finished z10 in 1s cpu:2s avg:2, now starting z11
0:01:41 INF [mbtiles:write] - Finished z11 in 1s cpu:3s avg:2, now starting z12
0:01:43 INF [mbtiles] -  features: [ 354k   7%  33k/s ] 270M  tiles: [ 140k  13k/s ] 10M  
    cpus: 1.9 gc:  7% heap: 2.1G/4.2G direct: 52M postGC: 1.2G
    read( 2%) -> (214/217) -> encode(57% 60%) -> (215/216) -> write( 4%)
    last tile: 12/1456/1661 (z12 35.56138277053833%) https://www.openstreetmap.org/#map=12/32.17561/-52.03125
0:01:44 INF [mbtiles:write] - Finished z12 in 3s cpu:5s avg:2, now starting z13
0:01:53 INF [mbtiles] -  features: [ 781k  15%  42k/s ] 270M  tiles: [ 291k  15k/s ] 13M  
    cpus: 2 gc:  3% heap: 1.6G/4.2G direct: 52M postGC: 1.3G
    read( 0%) -> (214/217) -> encode(63% 59%) -> (215/216) -> write( 3%)
    last tile: 13/2471/3047 (z13 30.171239376068115%) https://www.openstreetmap.org/#map=13/41.80408/-71.41113
0:01:56 INF [mbtiles:write] - Finished z13 in 13s cpu:25s avg:2, now starting z14
0:02:03 INF [mbtiles] -  features: [ 4.9M  94% 420k/s ] 270M  tiles: [ 3.6M 331k/s ] 99M  
    cpus: 1.9 gc:  2% heap: 2.5G/4.2G direct: 52M postGC: 1.3G
    read( 6%) ->   (0/217) -> encode(52% 52%) -> (215/216) -> write(62%)
    last tile: 14/6960/6125 (z14 42.48429015278816%) https://www.openstreetmap.org/#map=14/41.29432/-27.07031
0:02:04 DEB [mbtiles:write] - Shallow tiles written: 4,115,450
0:02:04 DEB [mbtiles:write] - Tile data written: 21,316 (99% omitted)
0:02:04 DEB [mbtiles:write] - Unique tile hashes: 11,290
0:02:04 INF [mbtiles:write] - Finished z14 in 8s cpu:13s avg:1.8
0:02:04 INF [mbtiles] -  features: [ 5.2M 100% 289k/s ] 270M  tiles: [ 4.1M 482k/s ] 108M 
    cpus: 1.3 gc:  0% heap: 2.9G/4.2G direct: 52M postGC: 1.3G
    read( -%) ->   (0/217) -> encode( -%  -%) ->   (0/216) -> write( -%)
    last tile: 14/7380/5985 (z14 45.04781886935234%) https://www.openstreetmap.org/#map=14/43.56447/-17.84180
0:02:04 DEB [mbtiles] - Tile stats:
0:02:04 DEB [mbtiles] - z0 avg:7.9k max:7.9k
0:02:04 DEB [mbtiles] - z1 avg:4k max:4k
0:02:04 DEB [mbtiles] - z2 avg:9.4k max:9.4k
0:02:04 DEB [mbtiles] - z3 avg:3.9k max:6.4k
0:02:04 DEB [mbtiles] - z4 avg:1.6k max:4.6k
0:02:04 DEB [mbtiles] - z5 avg:1.4k max:8.1k
0:02:04 DEB [mbtiles] - z6 avg:973 max:22k
0:02:04 DEB [mbtiles] - z7 avg:751 max:55k
0:02:04 DEB [mbtiles] - z8 avg:394 max:112k
0:02:04 DEB [mbtiles] - z9 avg:270 max:278k
0:02:04 DEB [mbtiles] - z10 avg:154 max:233k
0:02:04 DEB [mbtiles] - z11 avg:102 max:132k
0:02:04 DEB [mbtiles] - z12 avg:83 max:119k
0:02:04 DEB [mbtiles] - z13 avg:71 max:109k
0:02:04 DEB [mbtiles] - z14 avg:67 max:266k
0:02:04 DEB [mbtiles] - all avg:69 max:0
0:02:04 DEB [mbtiles] -  # features: 5,293,577
0:02:04 DEB [mbtiles] -     # tiles: 4,115,450
0:02:04 INF [mbtiles] - Finished in 32s cpu:1m gc:2s avg:1.9
0:02:04 INF [mbtiles] -   read    1x(3% 0.9s wait:29s)
0:02:04 INF [mbtiles] -   encode  2x(54% 17s wait:4s done:1s)
0:02:04 INF [mbtiles] -   write   1x(25% 8s wait:21s)
0:02:04 INF - Finished in 2m4s cpu:3m34s gc:5s avg:1.7
0:02:04 INF - FINISHED!
0:02:04 INF - 
0:02:04 INF - ----------------------------------------
0:02:04 INF - 	overall          2m4s cpu:3m34s gc:5s avg:1.7
0:02:04 INF - 	lake_centerlines 2s cpu:4s avg:1.8
0:02:04 INF - 	  read     1x(77% 2s)
0:02:04 INF - 	  process  2x(11% 0.3s wait:2s)
0:02:04 INF - 	  write    1x(0% 0s wait:2s)
0:02:04 INF - 	water_polygons   28s cpu:48s gc:3s avg:1.7
0:02:04 INF - 	  read     1x(58% 16s wait:3s)
0:02:04 INF - 	  process  2x(28% 8s wait:13s)
0:02:04 INF - 	  write    1x(2% 0.6s wait:27s)
0:02:04 INF - 	natural_earth    10s cpu:16s avg:1.6
0:02:04 INF - 	  read     1x(87% 9s sys:1s)
0:02:04 INF - 	  process  2x(20% 2s wait:9s)
0:02:04 INF - 	  write    1x(0% 0s wait:10s)
0:02:04 INF - 	osm_pass1        5s cpu:7s avg:1.5
0:02:04 INF - 	  read     1x(3% 0.1s wait:4s)
0:02:04 INF - 	  parse    1x(53% 2s wait:1s)
0:02:04 INF - 	  process  1x(35% 2s wait:2s)
0:02:04 INF - 	osm_pass2        32s cpu:1m4s avg:2
0:02:04 INF - 	  read     1x(0% 0.1s wait:16s done:17s)
0:02:04 INF - 	  process  2x(77% 25s)
0:02:04 INF - 	  write    1x(1% 0.4s wait:32s)
0:02:04 INF - 	boundaries       0s cpu:0.1s avg:1.6
0:02:04 INF - 	sort             3s cpu:3s avg:1.2
0:02:04 INF - 	  worker  1x(74% 2s)
0:02:04 INF - 	mbtiles          32s cpu:1m gc:2s avg:1.9
0:02:04 INF - 	  read    1x(3% 0.9s wait:29s)
0:02:04 INF - 	  encode  2x(54% 17s wait:4s done:1s)
0:02:04 INF - 	  write   1x(25% 8s wait:21s)
0:02:04 INF - ----------------------------------------
0:02:04 INF - 	features	270MB
0:02:04 INF - 	mbtiles	108MB
-rw-r--r-- 1 runner docker 55M Jul 11 06:58 run.jar

msbarry · 2022-06-10T10:13:00Z

Thanks for implementing this! A couple of concerns I'll want to mitigate before merging this:

Any areas within planetiler that assume tiles are ordered by z asc, x asc, y desc - I think there's a few but will need to dig a bit more - the one testSkipFill failing test might be indicative of something larger not covered by other tests.
Test performance impact for tile generation - we could write a microbenchmark for TileCoord encode/decode but I'd probably rather just test generating on a large extract or the planet and look at overall difference in performance.
Test difference in mbtiles read performance - you can give two mbtiles files to BenchmarkMbtilesRead and it reports the qps difference between the two, but reads tiles randomly - I assume we'd need something different to simulate more realistic traffic?

Also, do you have a link to the C++ code you ported? Want to check if there's any license/attribution requirements.

msbarry · 2022-06-10T10:15:51Z

Also for the "Check formatting" failure - if you run ./scripts/format.sh to autoformat the code and push that result it should fix the issue. I've got some instructions in CONTRIBUTING for setting up autoformat in a couple of IDEs if you were interested in that as well.

bdon · 2022-06-10T10:46:28Z

Thanks, I'll get started on these tasks. The implementation is part of my own C++ library which is not on GitHub.

msbarry · 2022-06-20T10:45:20Z

Took a quick look for potential issues I can see with tiles not ordered by z/x/y:

the previous ordering matches the sqlite index order exactly, it seemed like that would be optimal for building the index (size and speed) - to test that theory, could you post timing and size summaries printed at the end or a planet run for old z/x/y ordering and hilbert ordering?
MbtilesWriter.tileEncoder skips re-encoding/computing some things if the previous tile is the same as the current tile - for z/x/y this mean processing columns of filled polygons sequentially. For hilbert ordering that should be fine too since sequential tiles are adjacent?
Progress logging during mbtiles writes expects we process all z0 tiles, then z1, then z2 - that should still be the case for hilbert ordering?
MbtilesWriter.getLastTileLogDetails logs progress during mbtiles writes prints % complete for each zoom level, and a osm link to what part of the world it's working on so you can debug if things are slow. The OSM link should be more helpful with hilbert ordering since it's working on a part of the world instead of vertical slice. Is there a way to get a rough idea of % complete within a zoom level though? The previous logic used (100 * (lastTile.x() + 1 - zMinX)) / (zMaxX - zMinX) since we work in order across X columns.

If there's any reason we still might want z/x/y ordering then it might make sense to extract the ordering into a strategy you could swap out with a command-line flag. If there are no drawbacks of hilbert over z/x/y though then we don't need that.

Nothing before tile writing should care about tile ordering.

bdon · 2022-06-20T11:05:38Z

will start a new run tonight to get clean log output and post here, to address the rest:

for skipping encoding/re-encoding, the hilbert ordering should have a significant advantage over ZXY - the maximum single continuous run in ZXY is theoretically 16384 at z14, while the maximum single Hilbert segment for ocean could be 10-20x longer (need to verify this empirically)
Yes, the ordering of the tiles is Z0 first, then Z1, then Z2. There are alternate Hilbert designs that don't preserve this like the S2 Cell for a single Face, those require a fixed size of 64 bits but have much easier parent/child calculation (a parent is a binary prefix). For this Hilbert design: to find the parent of a given TileId you need to translate to ZXY, find Z/2, floor(x),floor(y) and then translate back to TileId.
Yes, the technique implemented above is really a combination of two parameters (zoom_level, position_on_level), so the fraction completed on a Z is (position_on_level / (2^zoom_level)^2)

planetiler-core/src/test/java/com/onthegomap/planetiler/PlanetilerTests.java

msbarry · 2022-06-20T12:26:58Z

Re: performance:

for encoding, it looks like there are several places that construct TileCoord instances just to store features in a map but don't actually care about the encoded int value. Only FeatureGroup.encodeKey accesses the encoded value outside of tests
for decoding, we only decode the value from FeatureGroup.TileFeatures constructor once per output tile

So it might make sense to just compute the encoded value on-demand instead of storing on the TileCoord instance, or computing it lazily now that it's not "free"

planetiler-core/src/main/java/com/onthegomap/planetiler/geo/TileCoord.java

bdon · 2022-06-22T05:57:35Z

run of planetiler.jar (Jun 5 release) 16cores

java -Xmx60g -jar planetiler.jar --area=planet --bounds=planet --download --download-threads=10 --download-chunk-size-mb=1000 --fetch-wikidata --mbtiles=output.mbtiles --nodemap-type=array --storage=mmap --force

2:56:05 DEB [mbtiles] - Tile stats:
2:56:05 DEB [mbtiles] - z0 avg:76k max:76k
2:56:05 DEB [mbtiles] - z1 avg:172k max:194k
2:56:05 DEB [mbtiles] - z2 avg:680k max:1.1M
2:56:05 DEB [mbtiles] - z3 avg:221k max:1M
2:56:05 DEB [mbtiles] - z4 avg:79k max:744k
2:56:05 DEB [mbtiles] - z5 avg:40k max:628k
2:56:05 DEB [mbtiles] - z6 avg:18k max:421k
2:56:05 DEB [mbtiles] - z7 avg:17k max:764k
2:56:05 DEB [mbtiles] - z8 avg:9.9k max:579k
2:56:05 DEB [mbtiles] - z9 avg:6.6k max:502k
2:56:05 DEB [mbtiles] - z10 avg:2.9k max:323k
2:56:05 DEB [mbtiles] - z11 avg:1.4k max:188k
2:56:05 DEB [mbtiles] - z12 avg:772 max:239k
2:56:05 DEB [mbtiles] - z13 avg:411 max:267k
2:56:05 DEB [mbtiles] - z14 avg:366 max:1.8M
2:56:05 DEB [mbtiles] - all avg:425 max:0
2:56:05 DEB [mbtiles] -  # features: 2,955,775,444
2:56:05 DEB [mbtiles] -     # tiles: 267,718,292
2:56:05 INF [mbtiles] - Finished in 1h17m9s cpu:18h15m38s gc:6m14s avg:14.2
2:56:05 INF [mbtiles] -   read    1x(12% 9m24s sys:2m36s wait:56m7s done:4s)
2:56:05 INF [mbtiles] -   encode 15x(86% 1h6m36s sys:15s wait:3m38s done:4s)
2:56:05 INF [mbtiles] -   write   1x(16% 12m11s sys:1m58s wait:58m19s)
2:56:06 INF - Finished in 2h56m6s cpu:38h56m49s gc:11m54s avg:13.3
2:56:06 INF - FINISHED!
2:56:06 INF -
2:56:06 INF - ----------------------------------------
2:56:06 INF -   overall          2h56m6s cpu:38h56m49s gc:11m54s avg:13.3
2:56:06 INF -   wikidata         5m14s cpu:1h18m42s gc:7s avg:15.1
2:56:06 INF -     pbf     1x(70% 3m39s sys:2m11s wait:1m15s done:9s)
2:56:06 INF -     filter 15x(93% 4m51s sys:1s wait:5s done:9s)
2:56:06 INF -     fetch   1x(1% 4s wait:5m2s done:8s)
2:56:06 INF -   lake_centerlines 2s cpu:7s avg:3.8
2:56:06 INF -     read     1x(84% 1s)
2:56:06 INF -     process 15x(5% 0.1s wait:2s)
2:56:06 INF -     write    1x(0% 0s wait:2s)
2:56:06 INF -   water_polygons   1m10s cpu:16m5s gc:3s avg:13.7
2:56:06 INF -     read     1x(22% 15s wait:48s done:2s)
2:56:06 INF -     process 15x(82% 57s wait:8s)
2:56:06 INF -     write    1x(37% 26s sys:8s wait:43s)
2:56:06 INF -   natural_earth    5s cpu:18s avg:3.3
2:56:06 INF -     read     1x(91% 5s)
2:56:06 INF -     process 15x(9% 0.5s wait:5s)
2:56:06 INF -     write    1x(0% 0s wait:5s)
2:56:06 INF -   osm_pass1        5m47s cpu:1h24m18s gc:8s avg:14.6
2:56:06 INF -     read     1x(49% 2m49s sys:1m42s wait:2m16s done:2s)
2:56:06 INF -     process 15x(91% 5m18s sys:8s block:6s wait:12s)
2:56:06 INF -   osm_pass2        1h13m21s cpu:15h48m56s gc:3m9s avg:12.9
2:56:06 INF -     read     1x(3% 2m14s sys:1m28s wait:1h10m14s done:36s)
2:56:06 INF -     process 15x(80% 58m52s sys:1m47s block:6s wait:6s)
2:56:06 INF -     write    1x(27% 19m59s sys:4m1s wait:51m37s)
2:56:06 INF -   boundaries       10s cpu:15s avg:1.4
2:56:06 INF -   sort             12m53s cpu:1h52m9s gc:2m11s avg:8.7
2:56:06 INF -     worker 16x(19% 2m27s sys:14s wait:2m11s done:13s)
2:56:06 INF -   mbtiles          1h17m9s cpu:18h15m38s gc:6m14s avg:14.2
2:56:06 INF -     read    1x(12% 9m24s sys:2m36s wait:56m7s done:4s)
2:56:06 INF -     encode 15x(86% 1h6m36s sys:15s wait:3m38s done:4s)
2:56:06 INF -     write   1x(16% 12m11s sys:1m58s wait:58m19s)
2:56:06 INF - ----------------------------------------
2:56:06 INF -   features        208GB
2:56:06 INF -   mbtiles 81GB

CONTRIBUTING.md

bdon · 2022-06-22T12:03:14Z

Branch planet run output:

2:57:24 DEB [mbtiles] - Tile stats:
2:57:24 DEB [mbtiles] - z0 avg:76k max:76k
2:57:24 DEB [mbtiles] - z1 avg:172k max:194k
2:57:24 DEB [mbtiles] - z2 avg:680k max:1.1M
2:57:24 DEB [mbtiles] - z3 avg:221k max:1M
2:57:24 DEB [mbtiles] - z4 avg:79k max:744k
2:57:24 DEB [mbtiles] - z5 avg:40k max:628k
2:57:24 DEB [mbtiles] - z6 avg:18k max:421k
2:57:24 DEB [mbtiles] - z7 avg:17k max:764k
2:57:24 DEB [mbtiles] - z8 avg:9.9k max:579k
2:57:24 DEB [mbtiles] - z9 avg:6.6k max:502k
2:57:24 DEB [mbtiles] - z10 avg:2.9k max:323k
2:57:24 DEB [mbtiles] - z11 avg:1.4k max:188k
2:57:24 DEB [mbtiles] - z12 avg:772 max:239k
2:57:24 DEB [mbtiles] - z13 avg:411 max:267k
2:57:24 DEB [mbtiles] - z14 avg:366 max:1.8M
2:57:24 DEB [mbtiles] - all avg:425 max:0
2:57:24 DEB [mbtiles] -  # features: 2,955,775,550
2:57:24 DEB [mbtiles] -     # tiles: 267,718,292
2:57:24 INF [mbtiles] - Finished in 1h17m19s cpu:18h23m10s gc:5m33s avg:14.3
2:57:24 INF [mbtiles] -   read    1x(13% 9m41s sys:2m39s wait:56m34s done:5s)
2:57:24 INF [mbtiles] -   encode 15x(86% 1h6m37s sys:22s wait:4m6s done:5s)
2:57:24 INF [mbtiles] -   write   1x(18% 13m56s sys:2m46s wait:59m18s)
2:57:24 INF - Finished in 2h57m24s cpu:38h58m1s gc:10m56s avg:13.2
2:57:24 INF - FINISHED!
2:57:24 INF -
2:57:24 INF - ----------------------------------------
2:57:24 INF -   overall          2h57m24s cpu:38h58m1s gc:10m56s avg:13.2
2:57:24 INF -   wikidata         5m16s cpu:1h18m16s gc:7s avg:14.8
2:57:24 INF -     pbf     1x(75% 3m57s sys:2m32s wait:57s done:9s)
2:57:24 INF -     filter 15x(91% 4m48s sys:1s wait:9s done:9s)
2:57:24 INF -     fetch   1x(1% 4s wait:5m5s done:8s)
2:57:24 INF -   lake_centerlines 2s cpu:6s avg:3.5
2:57:24 INF -     read     1x(83% 1s)
2:57:24 INF -     process 15x(4% 0.1s wait:2s)
2:57:24 INF -     write    1x(0% 0s wait:2s)
2:57:24 INF -   water_polygons   1m11s cpu:15m52s gc:3s avg:13.5
2:57:24 INF -     read     1x(20% 14s wait:50s done:3s)
2:57:24 INF -     process 15x(80% 56s wait:9s)
2:57:24 INF -     write    1x(38% 27s sys:8s wait:42s)
2:57:24 INF -   natural_earth    5s cpu:21s avg:4
2:57:24 INF -     read     1x(88% 5s)
2:57:24 INF -     process 15x(13% 0.7s wait:4s)
2:57:24 INF -     write    1x(0% 0s wait:5s)
2:57:24 INF -   osm_pass1        5m52s cpu:1h23m3s gc:9s avg:14.1
2:57:24 INF -     read     1x(53% 3m8s sys:2m1s wait:1m45s done:2s)
2:57:24 INF -     process 15x(88% 5m11s sys:9s block:6s wait:22s)
2:57:24 INF -   osm_pass2        1h14m21s cpu:15h51m gc:2m57s avg:12.8
2:57:24 INF -     read     1x(3% 2m28s sys:1m40s wait:1h10m53s done:37s)
2:57:24 INF -     process 15x(80% 59m7s sys:1m58s block:9s wait:9s)
2:57:24 INF -     write    1x(28% 20m46s sys:4m8s wait:51m49s)
2:57:24 INF -   boundaries       10s cpu:15s avg:1.4
2:57:24 INF -   sort             12m54s cpu:1h45m34s gc:2m5s avg:8.2
2:57:24 INF -     worker 16x(18% 2m16s sys:14s wait:2m33s done:15s)
2:57:24 INF -   mbtiles          1h17m19s cpu:18h23m10s gc:5m33s avg:14.3
2:57:24 INF -     read    1x(13% 9m41s sys:2m39s wait:56m34s done:5s)
2:57:24 INF -     encode 15x(86% 1h6m37s sys:22s wait:4m6s done:5s)
2:57:24 INF -     write   1x(18% 13m56s sys:2m46s wait:59m18s)
2:57:24 INF - ----------------------------------------
2:57:24 INF -   features        208GB
2:57:24 INF -   mbtiles 82GB

bdon · 2022-06-22T12:21:15Z

MBTiles benchmark for BRANCH:

worker@Ubuntu-2004-focal-64-minimal:~$ java -cp planetiler-dist-0.5-SNAPSHOT-with-deps.jar com.onthegomap.planetiler.benchmarks.BenchmarkMbtilesRead --bench_mbtiles=output.mbtiles
0:00:00 DEB - argument: bench_repetitions=10 (number of repetitions)
0:00:00 DEB - argument: bench_nr_tile_reads=500000 (number of tiles to read)
0:00:00 DEB - argument: bench_pre_warms=3 (number of pre warm runs)
0:00:00 DEB - argument: bench_mbtiles=output.mbtiles (the mbtiles file to read from)
0:08:41 INF - working on output.mbtiles
0:09:24 INF - readOperationsPerSecondStats: DoubleSummaryStatistics{count=10, sum=1603208.694870, min=159937.217605, average=160320.869487, max=160890.394421}
0:09:24 INF - diffs

sqlite3_analyzer output for BRANCH:

https://gist.github.com/bdon/0022346493786e58e12b4300bfd443ee

bdon · 2022-06-23T05:51:44Z

MBTiles benchmark for MAIN:

java -cp planetiler-dist-0.5-SNAPSHOT-with-deps.jar com.onthegomap.planetiler.benchmarks.BenchmarkMbtilesRead --bench_mbtiles=output.mbtiles
0:00:00 DEB - argument: bench_repetitions=10 (number of repetitions)
0:00:00 DEB - argument: bench_nr_tile_reads=500000 (number of tiles to read)
0:00:00 DEB - argument: bench_pre_warms=3 (number of pre warm runs)
0:00:00 DEB - argument: bench_mbtiles=output.mbtiles (the mbtiles file to read from)
0:02:32 INF - working on output.mbtiles
0:03:14 INF - readOperationsPerSecondStats: DoubleSummaryStatistics{count=10, sum=1629460.846341, min=162661.918839, average=162946.084634, max=163245.337260}
0:03:14 INF - diffs

sqlite3_analyzer output for MAIN:

https://gist.github.com/bdon/fe0111a676d8ba6bf432538cd6b69344

msbarry · 2022-06-23T09:35:45Z

Thanks @bdon, looks like hilbert ordering is a tiny bit slower on random reads but that might not be a very accurate test. Seems like hilbert ordering is meant to help with more realistic traffic distribution on machine with less RAM. What have you done in the past to test pmtiles reads with a more realistic traffic distribution?

bdon · 2022-06-23T16:26:57Z

I've only seen the gains from the ordering in serving latency from production traffic, but a reproducible benchmark would be good to add to https://github.com/bdon/TileSiege ; I think if we could get an anonymized day of real traffic that would be a more effective benchmark for testing query latency. It does look like the hilbert order also ends up with less full pages so the db ends up larger, but I wasn't running it with the vacuum DB option (is that the default?)

msbarry · 2022-06-24T11:00:20Z

Tested on a c6gd.metal instance (64 cpu/128GB RAM):

zxy (main) hilbert (branch)

0:36:40 INF - 	overall          36m40s cpu:27h6m13s gc:2m45s avg:44.3
0:36:40 INF - 	lake_centerlines 2s cpu:8s avg:4.1
0:36:40 INF - 	  read     1x(86% 2s)
0:36:40 INF - 	  process 62x(1% 0s wait:2s)
0:36:40 INF - 	  write    2x(0% 0s wait:2s)
0:36:40 INF - 	water_polygons   35s cpu:20m35s gc:1s avg:35.2
0:36:40 INF - 	  read     1x(44% 16s sys:1s wait:16s done:2s)
0:36:40 INF - 	  process 62x(51% 18s wait:16s)
0:36:40 INF - 	  write    2x(61% 21s sys:3s wait:13s)
0:36:40 INF - 	natural_earth    9s cpu:25s avg:2.7
0:36:40 INF - 	  read     1x(92% 8s sys:1s)
0:36:40 INF - 	  process 62x(2% 0.2s wait:8s)
0:36:40 INF - 	  write    2x(0% 0s wait:9s)
0:36:40 INF - 	osm_pass1        1m33s cpu:1h23m42s gc:9s avg:54.1
0:36:40 INF - 	  read     1x(52% 48s sys:39s wait:28s done:6s)
0:36:40 INF - 	  process 63x(77% 1m11s sys:7s block:10s)
0:36:40 INF - 	osm_pass2        17m53s cpu:17h47m50s gc:1m32s avg:59.7
0:36:40 INF - 	  read     1x(6% 1m1s sys:47s wait:15m48s done:46s)
0:36:40 INF - 	  process 62x(84% 14m59s block:6s wait:22s)
0:36:40 INF - 	  write    2x(47% 8m19s sys:1m8s wait:7m22s)
0:36:40 INF - 	boundaries       18s cpu:23s avg:1.3
0:36:40 INF - 	sort             3m8s cpu:1h12m27s gc:15s avg:23.1
0:36:40 INF - 	  worker 30x(9% 17s sys:5s wait:2m2s done:15s)
0:36:40 INF - 	mbtiles          12m56s cpu:6h20m31s gc:47s avg:29.4
0:36:40 INF - 	  read    2x(25% 3m17s sys:54s wait:8m13s done:23s)
0:36:40 INF - 	  merge   1x(38% 4m52s sys:12s wait:7m12s done:11s)
0:36:40 INF - 	  encode 62x(40% 5m9s sys:9s wait:6m59s done:11s)
0:36:40 INF - 	  write   1x(84% 10m50s sys:1m12s wait:1m3s)
0:36:40 INF - ----------------------------------------
0:36:40 INF - 	features	185GB
0:36:40 INF - 	mbtiles	79GB

0:39:56 INF - 	overall          39m56s cpu:27h48m38s gc:2m50s avg:41.8
0:39:56 INF - 	lake_centerlines 2s cpu:8s avg:3.8
0:39:56 INF - 	  read     1x(86% 2s)
0:39:56 INF - 	  process 62x(1% 0s wait:2s)
0:39:56 INF - 	  write    2x(0% 0s wait:2s)
0:39:56 INF - 	water_polygons   35s cpu:21m20s gc:1s avg:36.3
0:39:56 INF - 	  read     1x(45% 16s sys:2s wait:16s done:2s)
0:39:56 INF - 	  process 62x(53% 19s sys:1s wait:15s)
0:39:56 INF - 	  write    2x(61% 21s sys:3s wait:13s)
0:39:56 INF - 	natural_earth    9s cpu:25s avg:2.8
0:39:56 INF - 	  read     1x(91% 8s sys:1s)
0:39:56 INF - 	  process 62x(2% 0.2s wait:8s)
0:39:56 INF - 	  write    2x(0% 0s wait:9s)
0:39:56 INF - 	osm_pass1        1m36s cpu:1h26m54s gc:10s avg:54.2
0:39:56 INF - 	  read     1x(51% 49s sys:40s wait:28s done:6s)
0:39:56 INF - 	  process 63x(75% 1m12s sys:5s block:10s)
0:39:56 INF - 	osm_pass2        18m23s cpu:18h19m47s gc:1m39s avg:59.8
0:39:56 INF - 	  read     1x(6% 1m7s sys:52s wait:16m12s done:46s)
0:39:56 INF - 	  process 62x(82% 15m8s block:6s wait:11s done:8s)
0:39:56 INF - 	  write    2x(43% 7m55s sys:1m12s wait:8m17s)
0:39:56 INF - 	boundaries       18s cpu:22s avg:1.3
0:39:56 INF - 	sort             3m9s cpu:1h12m41s gc:15s avg:23
0:39:56 INF - 	  worker 30x(9% 17s sys:6s wait:2m4s done:13s)
0:39:56 INF - 	mbtiles          15m38s cpu:6h26m48s gc:44s avg:24.8
0:39:56 INF - 	  read    2x(19% 3m sys:50s wait:10m33s done:38s)
0:39:56 INF - 	  merge   1x(29% 4m34s sys:11s wait:10m4s done:14s)
0:39:56 INF - 	  encode 62x(34% 5m15s sys:13s block:3s wait:9m24s done:14s)
0:39:56 INF - 	  write   1x(85% 13m16s sys:2m12s wait:1m26s)
0:39:56 INF - ----------------------------------------
0:39:56 INF - 	features	185GB
0:39:56 INF - 	mbtiles	80GB

And also tested reads on the c6gd.metal instance with 1 million real tile requests from onthegomap:

$ java -Xmx4g -cp planetiler-hilbert.jar com.onthegomap.planetiler.benchmarks.BenchmarkMbtilesRead -bench-tiles-path=tiles.csv --bench-mbtiles=data/zxy.mbtiles,data/hilbert.mbtiles --bench-nr-tile-reads=2000000
0:00:00 DEB - argument: bench_repetitions=10 (number of repetitions)
0:00:00 DEB - argument: bench_tiles_path=tiles.csv (path to z,x,y tiles)
0:00:00 DEB - argument: bench_nr_tile_reads=2000000 (number of tiles to read)
0:00:00 DEB - argument: bench_pre_warms=3 (number of pre warm runs)
0:00:00 DEB - argument: bench_mbtiles=data/zxy.mbtiles,data/hilbert.mbtiles (the mbtiles file to read from)
0:00:05 INF - working on data/zxy.mbtiles
0:01:18 INF - readOperationsPerSecondStats: DoubleSummaryStatistics{count=10, sum=3663908.498654, min=363275.717243, average=366390.849865, max=368180.724534}
0:01:18 INF - working on data/hilbert.mbtiles
0:02:29 INF - readOperationsPerSecondStats: DoubleSummaryStatistics{count=10, sum=3672214.184589, min=362048.229926, average=367221.418459, max=369536.268366}
0:02:29 INF - diffs
0:02:29 INF - "data/zxy.mbtiles" to "data/hilbert.mbtiles": avg read operations per second improved by 0.22668922923372747%

And a memory-constrained c6gd.xlarge instance (4 cpu/8GB RAM):

$ java -Xmx4g -cp planetiler.jar com.onthegomap.planetiler.benchmarks.BenchmarkMbtilesRead -bench-tiles-path=tiles.csv --bench-mbtiles=zxy.mbtiles,hilbert.mbtiles --bench-nr-tile-reads=2000000
0:00:00 DEB - argument: bench_repetitions=10 (number of repetitions)
0:00:00 DEB - argument: bench_tiles_path=tiles.csv (path to z,x,y tiles)
0:00:00 DEB - argument: bench_nr_tile_reads=2000000 (number of tiles to read)
0:00:00 DEB - argument: bench_pre_warms=3 (number of pre warm runs)
0:00:00 DEB - argument: bench_mbtiles=zxy.mbtiles,hilbert.mbtiles (the mbtiles file to read from)
0:00:05 INF - working on zxy.mbtiles
0:01:20 INF - readOperationsPerSecondStats: DoubleSummaryStatistics{count=10, sum=3545666.701217, min=353559.481729, average=354566.670122, max=358342.863245}
0:01:20 INF - working on hilbert.mbtiles
0:02:33 INF - readOperationsPerSecondStats: DoubleSummaryStatistics{count=10, sum=3554540.411371, min=354136.382240, average=355454.041137, max=356223.100484}
0:02:33 INF - diffs
0:02:33 INF - "zxy.mbtiles" to "hilbert.mbtiles": avg read operations per second improved by 0.25026915674310146%

…o zoom 15 [onthegomap#98, onthegomap#128]

sonarcloud · 2022-07-11T07:00:35Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

38.7% Coverage
0.0% Duplication

bdon · 2022-07-20T08:31:38Z

New TMS ordering run on same machine:

2:57:55 DEB [mbtiles] - Tile stats:
2:57:55 DEB [mbtiles] - z0 avg:76k max:76k
2:57:55 DEB [mbtiles] - z1 avg:172k max:194k
2:57:55 DEB [mbtiles] - z2 avg:680k max:1.1M
2:57:55 DEB [mbtiles] - z3 avg:221k max:1M
2:57:55 DEB [mbtiles] - z4 avg:79k max:744k
2:57:55 DEB [mbtiles] - z5 avg:40k max:628k
2:57:55 DEB [mbtiles] - z6 avg:18k max:421k
2:57:55 DEB [mbtiles] - z7 avg:17k max:764k
2:57:55 DEB [mbtiles] - z8 avg:9.9k max:579k
2:57:55 DEB [mbtiles] - z9 avg:6.6k max:502k
2:57:55 DEB [mbtiles] - z10 avg:2.9k max:323k
2:57:55 DEB [mbtiles] - z11 avg:1.4k max:188k
2:57:55 DEB [mbtiles] - z12 avg:772 max:239k
2:57:55 DEB [mbtiles] - z13 avg:411 max:267k
2:57:55 DEB [mbtiles] - z14 avg:366 max:1.8M
2:57:55 DEB [mbtiles] - all avg:425 max:0
2:57:55 DEB [mbtiles] -  # features: 2,955,775,537
2:57:55 DEB [mbtiles] -     # tiles: 267,718,292
2:57:55 INF [mbtiles] - Finished in 1h17m38s cpu:18h29m38s gc:5m37s avg:14.3
2:57:55 INF [mbtiles] -   read    1x(12% 9m32s sys:2m37s wait:56m29s done:4s)
2:57:55 INF [mbtiles] -   encode 15x(86% 1h7m5s sys:21s wait:2m41s done:4s)
2:57:55 INF [mbtiles] -   write   1x(16% 12m31s sys:2m11s wait:59m16s)
2:57:55 INF - Finished in 2h57m56s cpu:39h23m56s gc:11m39s avg:13.3
2:57:55 INF - FINISHED!
2:57:55 INF -
2:57:55 INF - ----------------------------------------
2:57:55 INF -   overall          2h57m56s cpu:39h23m56s gc:11m39s avg:13.3
2:57:55 INF -   wikidata         5m27s cpu:1h16m31s gc:9s avg:14
2:57:55 INF -     pbf     1x(70% 3m50s sys:2m27s wait:37s done:10s)
2:57:55 INF -     filter 15x(86% 4m41s sys:1s wait:9s done:9s)
2:57:55 INF -     fetch   1x(1% 3s wait:5m17s done:8s)
2:57:55 INF -   lake_centerlines 1s cpu:6s avg:5.1
2:57:55 INF -     read     1x(85% 1s)
2:57:55 INF -     process 15x(6% 0.1s wait:1s)
2:57:55 INF -     write    1x(1% 0s wait:1s)
2:57:55 INF -   water_polygons   1m12s cpu:16m10s gc:3s avg:13.4
2:57:55 INF -     read     1x(20% 14s wait:51s done:3s)
2:57:55 INF -     process 15x(80% 58s wait:9s)
2:57:55 INF -     write    1x(38% 27s sys:8s wait:43s)
2:57:55 INF -   natural_earth    6s cpu:21s avg:3.6
2:57:55 INF -     read     1x(90% 5s)
2:57:55 INF -     process 15x(12% 0.7s wait:5s)
2:57:55 INF -     write    1x(0% 0s wait:6s)
2:57:55 INF -   osm_pass1        6m3s cpu:1h24m11s gc:9s avg:13.9
2:57:55 INF -     read     1x(51% 3m7s sys:2m1s wait:1m31s done:1s)
2:57:55 INF -     process 15x(87% 5m15s sys:9s block:6s wait:13s)
2:57:55 INF -   osm_pass2        1h17m2s cpu:16h13m31s gc:3m20s avg:12.6
2:57:55 INF -     read     1x(3% 2m23s sys:1m37s wait:1h13m35s done:38s)
2:57:55 INF -     process 15x(78% 1h22s sys:1m51s block:6s wait:6s)
2:57:55 INF -     write    1x(26% 20m5s sys:4m6s wait:54m35s)
2:57:55 INF -   boundaries       11s cpu:15s avg:1.4
2:57:55 INF -   sort             9m59s cpu:1h42m47s gc:2m19s avg:10.3
2:57:55 INF -     worker 16x(11% 1m9s sys:17s wait:4m29s done:10s)
2:57:55 INF -   mbtiles          1h17m38s cpu:18h29m38s gc:5m37s avg:14.3
2:57:55 INF -     read    1x(12% 9m32s sys:2m37s wait:56m29s done:4s)
2:57:55 INF -     encode 15x(86% 1h7m5s sys:21s wait:2m41s done:4s)
2:57:55 INF -     write   1x(16% 12m31s sys:2m11s wait:59m16s)
2:57:55 INF - ----------------------------------------
2:57:55 INF -   features        208GB
2:57:55 INF -   mbtiles 81GB

sonarcloud · 2023-01-07T12:33:42Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

sonarcloud · 2023-01-08T05:53:29Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

msbarry mentioned this pull request Jun 20, 2022

Fix memoize bug #279

Merged

msbarry reviewed Jun 20, 2022

View reviewed changes

planetiler-core/src/test/java/com/onthegomap/planetiler/PlanetilerTests.java Outdated Show resolved Hide resolved

bdon force-pushed the tilecoord-hilbert branch from 7b6093e to 543e8d4 Compare June 21, 2022 07:32

bdon commented Jun 21, 2022

View reviewed changes

planetiler-core/src/main/java/com/onthegomap/planetiler/geo/TileCoord.java Outdated Show resolved Hide resolved

bdon force-pushed the tilecoord-hilbert branch from 8bd6bd1 to 558b660 Compare June 22, 2022 10:10

msbarry reviewed Jun 22, 2022

View reviewed changes

CONTRIBUTING.md Outdated Show resolved Hide resolved

bdon and others added 5 commits July 11, 2022 10:34

proof-of-concept for TileCoord hilbert curve sequence supporting up t…

6726e6d

…o zoom 15 [onthegomap#98, onthegomap#128]

benchmark

f88167b

add faster hilbert implementation and benchmark [onthegomap#128]

9110378

test z15 boundary cases

ffcb808

TileCoord.progressOnLevel for percent completion logging

448a7a6

bdon force-pushed the tilecoord-hilbert branch from 230727a to 448a7a6 Compare July 11, 2022 02:35

implement TMS ordering for TileCoord up to z15 [onthegomap#128]

b4cf3a1

bdon changed the title ~~TileCoord supports up to zoom 15 using Hilbert ordering~~ TileCoord supports up to zoom 15 using alternate ordering Jul 11, 2022

bdon added 2 commits July 11, 2022 14:50

attempt to fix javadoc

f24aafd

broken javadoc reference

30890b6

msbarry merged commit a50c438 into onthegomap:main Jul 24, 2022

msbarry mentioned this pull request Jul 24, 2022

Support zoom levels up to 15 #303

Merged

bdon deleted the tilecoord-hilbert branch January 13, 2023 03:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TileCoord supports up to zoom 15 using alternate ordering #266

TileCoord supports up to zoom 15 using alternate ordering #266

bdon commented Jun 8, 2022

github-actions bot commented Jun 8, 2022 •

edited

msbarry commented Jun 10, 2022

msbarry commented Jun 10, 2022

bdon commented Jun 10, 2022

msbarry commented Jun 20, 2022 •

edited

bdon commented Jun 20, 2022

msbarry commented Jun 20, 2022

bdon commented Jun 22, 2022

bdon commented Jun 22, 2022

bdon commented Jun 22, 2022 •

edited

bdon commented Jun 23, 2022

msbarry commented Jun 23, 2022

bdon commented Jun 23, 2022

msbarry commented Jun 24, 2022 •

edited

sonarcloud bot commented Jul 11, 2022

bdon commented Jul 20, 2022

sonarcloud bot commented Jan 7, 2023

sonarcloud bot commented Jan 8, 2023

TileCoord supports up to zoom 15 using alternate ordering #266

TileCoord supports up to zoom 15 using alternate ordering #266

Conversation

bdon commented Jun 8, 2022

github-actions bot commented Jun 8, 2022 • edited

msbarry commented Jun 10, 2022

msbarry commented Jun 10, 2022

bdon commented Jun 10, 2022

msbarry commented Jun 20, 2022 • edited

bdon commented Jun 20, 2022

msbarry commented Jun 20, 2022

bdon commented Jun 22, 2022

bdon commented Jun 22, 2022

bdon commented Jun 22, 2022 • edited

bdon commented Jun 23, 2022

msbarry commented Jun 23, 2022

bdon commented Jun 23, 2022

msbarry commented Jun 24, 2022 • edited

sonarcloud bot commented Jul 11, 2022

bdon commented Jul 20, 2022

sonarcloud bot commented Jan 7, 2023

sonarcloud bot commented Jan 8, 2023

github-actions bot commented Jun 8, 2022 •

edited

msbarry commented Jun 20, 2022 •

edited

bdon commented Jun 22, 2022 •

edited

msbarry commented Jun 24, 2022 •

edited