Create and write tileset tile by tile

[LorenzoMarnat]

Create and write the tileset tile by tile. Each tile is transformed (reprojected, translated, etc) separately. Once a tile is created, its content (.b3dm) is writen on the disk and deleted from the memory.

Also, the CityTiler now loads the geometries of the features tile by tile. The distribution of the features into tiles can be done with only the centroid of each feature. When dealing with huge amount of data, loading only the triangles of the current tile drastically reduce the memory usage.

This PR also comes with a rework of the LODs creation. The objectives were to:

• Create the LODs tile by tile (and not before the tileset creation)
• Clarify the LOA creation

See issue for more details

Note: py3dtiles needs to be uninstalled and reinstalled, else the write_as_json method of the TileSet class won't be found.

[jailln]

Niiice :) I have been testing it since your message in #58 but I came across the following error for big tilesets:

  File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Texture/atlas_node.py", line 46, in insert
    newNode = self.child[1].insert(img, building_id)
  [Previous line repeated 724 more times]
  File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Texture/atlas_node.py", line 107, in insert
    self.child[0].insert(img, building_id)
  File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Texture/atlas_node.py", line 107, in insert
    self.child[0].insert(img, building_id)
  File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Texture/atlas_node.py", line 59, in insert
    if self.rect.perfect_fits(img):
  File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/Texture/atlas_rectangle.py", line 52, in perfect_fits
    imageWidth, imageHeight = img.size
RecursionError: maximum recursion depth exceeded

I see that you added commits since then, I will try with this new version and let you know if I still hit this error.

[LorenzoMarnat]

I didn't get this error, but since with this PR the surfaces are split after the kd_tree distribution, there were 500 buildings per tile even when creating textured 3DTiles. Since each building has ~20 surfaces, we tried to create atlas with 500 * 20 images. I guess the Atlas algorithm can't handle that much images.

This commit should fix your problem. I had to limit the maxium number of buildings in the kd_tree to 25 when creating textures because having 10000+ images loaded was to much for the memory. With 25 buildings per tile, the number of surfaces (and images) is the same as before this PR.

[jailln]

Ok, thanks for the details.
I've tested it, the tiles gets written (with a steady RAM usage 🤩 ) but the tileset doesn't and I get the following error:

4095 / 4096 tiles created
Tileset created
Traceback (most recent call last):
  File "/home/vincent/Documents/dev/py3dtilers/venv/bin/citygml-tiler", line 33, in <module>
    sys.exit(load_entry_point('py3dtilers', 'console_scripts', 'citygml-tiler')())
  File "/home/vincent/Documents/dev/py3dtilers/py3dtilers/CityTiler/CityTiler.py", line 151, in main
    tileset.write_as_json(city_tiler.get_output_dir())
AttributeError: 'TileSet' object has no attribute 'write_as_json'

[LorenzoMarnat]

Sorry I forgot to mention it, but the method write_as_json has been added to py3dtiles just before the PR. You should uninstall py3dtiles then reinstall py3dtilers to get the last version of py3dtiles.

pip uninstall py3dtiles
pip install -e .

[jailln]

Oh ok, thanks! I restart the transformation

[jailln]

That did the trick thanks! The resulting 3D Tiles is rendered a bit faster (maybe due to the number of features per tile tat has been modified).
I tested it on a 8.5Go dataset, I will try on one of 24.23Go now and let you know if it still works.

[clementcolin]

Seems fine to me, great job !
Perhaps, we may need to update the doc somehow to explain this new way to create a tileset (one tile at a time).

Also, in the future, try to keep one change by PR so that it is easier to read. Some changes here are related to the new name "Feature", to the new way of tiling, and to clarification of LOA creation.

[jailln]

That did the trick thanks! The resulting 3D Tiles is rendered a bit faster (maybe due to the number of features per tile tat has been modified). I tested it on a 8.5Go dataset, I will try on one of 24.23Go now and let you know if it still works.

It works like a charm, thanks!