Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use rtree for large polygons #323

Closed
wants to merge 2 commits into from
Closed

Conversation

kleunen
Copy link
Contributor

@kleunen kleunen commented Sep 19, 2021

Use rtree only for polygons that cover large areas.

I actually got Africa converted with this on my 8G machine, it shows quite a significant saving in memory. It stores polygons in the rtree if the polygon covers more than 2 tiles at basezoom. It can also store linestrings, but I saw that this does impact the performance of the conversion.

@systemed
Copy link
Owner

Timings for Great Britain:

User time (seconds): 12276.29
System time (seconds): 154.09
Elapsed (wall clock) time (h:mm:ss or m:ss): 22:21.18
Maximum resident set size (kbytes): 14389332
File system inputs: 4653680
File system outputs: 2188184

So compared to #314 (comment), slightly smaller memory usage than master (which is what you'd hope, given that the biggest differences will be with polar regions etc.), but surprisingly quite a lot faster!

@kleunen
Copy link
Contributor Author

kleunen commented Sep 20, 2021

Hmm, only slightly smaller. It really depends on the area, on the geofabrik Africa extract, i see about 3G/4G lower memory consumption on my machine. So a huge difference.

I wonder if we can also optimize storing the linestring.

@systemed
Copy link
Owner

I wouldn't expect GB to be much smaller - there aren't many big polygons (apart from the coastline). Africa sounds like a big improvement.

One slightly odd rendering issue - this is a GB extract so I wouldn't necessarily expect Ireland to be rendered, but at some zoom levels, the tiles chosen for rendering are a bit weird:

Screenshot 2021-09-20 at 10 02 32

@kleunen
Copy link
Contributor Author

kleunen commented Sep 20, 2021

Yes, it is because the rtree polygons are now not included in calculating the to be rendered tiles anymore.

Maybe the bounds of the rtree should be included.

@kleunen
Copy link
Contributor Author

kleunen commented Sep 20, 2021

Have another try, the tiles are calculated from the clipping box now.
So this now always needs to be set when converting.

@systemed
Copy link
Owner

Clipping box looks good!

Similar memory usage to last time, time up slightly (but then it's generating more tiles so I'd expect that):

User time (seconds): 19092.60
System time (seconds): 161.80
Elapsed (wall clock) time (h:mm:ss or m:ss): 26:22.38
Maximum resident set size (kbytes): 14428496
File system inputs: 4653360
File system outputs: 2453648

@kleunen
Copy link
Contributor Author

kleunen commented Sep 21, 2021

The memory usage should not go up, the additional tiles which are generated now are mostly empty.

Maybe the trade-off when to store in rtree can be improved.

@kleunen
Copy link
Contributor Author

kleunen commented Sep 21, 2021

Have another try after optimizing things a bit :)

@systemed
Copy link
Owner

Very similar! Happy to run it on a different extract if you think it might be interesting.

User time (seconds): 19011.64
System time (seconds): 209.51
Elapsed (wall clock) time (h:mm:ss or m:ss): 26:51.59
Maximum resident set size (kbytes): 14428084
File system inputs: 4653616
File system outputs: 2458384

@kleunen
Copy link
Contributor Author

kleunen commented Sep 21, 2021

I guess it is difficult to compare now. Because both the number of tiles increased + the effect of the rtree.

@kleunen
Copy link
Contributor Author

kleunen commented Sep 22, 2021

The filesystem outputs is significantly increased, what does this number mean ?

@systemed
Copy link
Owner

systemed commented Sep 22, 2021

It's apparently "the total number of bytes written / 512". I'd expect it to be a bit greater given that the bbox is larger. (Not sure why "file system inputs" was 0 for master in #314 (comment) - that seems wrong given that it has to read the shapefiles and the .pbf.)

Currently running this branch on planet.osm.pbf - should have a result/timing some time tomorrow.

@systemed
Copy link
Owner

My attempt to run this on the planet was much slower than before (after 37 hours it was still on z11), but this might be because I was using the full extent of the planet, whereas previously I was using a bbox that excluded the polar regions. I'll try with some different areas and bboxes.

@kleunen
Copy link
Contributor Author

kleunen commented Sep 23, 2021

Yes. Maybe focus on the lower left quarter of the planet. So the region which includes south america and the south pole. On my machine this convert within couple of hours. And compare this PR with the current master.

image

@kleunen
Copy link
Contributor Author

kleunen commented Sep 23, 2021

On my Netherlands extract, not much difference between the two approaches:
Master:
Stored 114325890 nodes, 230175 ways, 79706 relations
Shape points: 0, lines: 0, polygons: 147
Generated points: 9776589, lines: 4456623, polygons: 12683656
Zoom level 14, writing tile 62189 of 62189
Memory used: 10231576

Filled the tileset with good things at netherlands.mbtiles

real 5m34.782s
user 59m40.483s
sys 0m53.007s

RTREE:
Stored 114325890 nodes, 230176 ways, 79706 relations
Shape points: 0, lines: 0, polygons: 147
Generated points: 9776589, lines: 4456623, polygons: 12683657
Zoom level 14, writing tile 63130 of 63130
Memory used: 10236976

Filled the tileset with good things at netherlands.mbtiles

real 5m40.357s
user 60m14.247s
sys 0m56.997s

@systemed
Copy link
Owner

I'm currently trying this again on the full planet with --bbox -180,-60,180,75... it's working through z11 (slowly!) at the moment.

@systemed
Copy link
Owner

After 37 hours (the time it took to generate the planet in #315) it was still going very slowly through z12 so I killed it.

I'm now running it with the rtree used for shapefiles but not for OSM-derived objects, to see if this makes a difference. Peak memory consumption is 115GB which is a handy improvement over #315.

@kleunen
Copy link
Contributor Author

kleunen commented Oct 23, 2021

Yes, I think it is really needed to try out what configuration reduces the memory in all cases. I do not fully understand under what condition the usage of the rtree can reduce the overall memory consumption. Maybe using the rtree only for the shapesfile is a better option.

It seems the existing approach with the tile_index is already quite an efficient approach. So it is difficult to improve on this.

@systemed
Copy link
Owner

I think the rule of thumb is that:

  • small objects are most efficiently stored with the tile index (i.e. one OutputObjectRef in a single tile)
  • large objects are most efficiently stored with the rtree (i.e. one OutputObjectRef+bbox which covers many tiles)

So the rtree makes particular sense for coastline polygons which are very big and can cover many tiles, particularly approaching the poles, where the spherical Mercator projection tends to stretch the y axis => more tiles. The coastline shapefile can potentially take up 26GB just in OutputObjectRefs, so this is a big deal.

There is a performance issue with the rtree approach. Even when restricting it to shapefiles only, certain areas are much slower to write than before, especially at z14, whereas others are still fast. I haven't figured out why yet. One possible reason:

  • with the current (tile index) approach, we only include the polygon in those z14 tiles which intersect the actual polygon:

current

  • with an rtree, we consider it in those z14 tiles which intersect its bounding box:

rtree

Consequently we're processing a big complex multipolygon for many tiles where we weren't doing so before.

If this is the case then there might be a few ways round it - e.g. storing a compressed list of affected tiles with each rtree entry (e.g. "row 0 has tiles in columns 163-189, row 1 has tiles in columns 164-188, row 2 has..." etc.), or subdividing the bboxes into multiple smaller bboxes each with their own rtree entry.

@kleunen
Copy link
Contributor Author

kleunen commented Oct 25, 2021

Maybe it is an idea to define the tileindex at multiple zoom levels.
When there is a block of 4 tiles covered at z14, this may possibly be stored as single entry at z13
When there is a block of 4 tiles covered at z13, this may possibly be stored as single entry at z12
...

I think with only a limited number of zoom levels (3 or 4), you can get a reduction in the memory usage of the tileindex.

@systemed
Copy link
Owner

systemed commented Oct 25, 2021

Yes, that could work!

I did also wonder whether adding an explicit ::intersects check (maybe only for shapefile/rtree-derived objects?) might help with the multipolygon issue above.

Results from --bbox -180,-60,180,75 with the rtree restricted to shapefile data only:

  • peak RAM c. 115GB (vs 131GB previously)
  • execution time 49hr48 (vs c. 37 hours previously)
  • planet.mbtiles size 78.7GB (vs 69GB previously) - don't know what's causing this
  • Fast shutdown #320 worked :)

@systemed
Copy link
Owner

Reworked and merged as #479.

@systemed systemed closed this Mar 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants