Loaders.gl 3D-tiled middleware

[jo-chemla]

I'm hoping this is the right place for such a discussion - this follows a dream thread I've made a while back regarding what could be called Ti3D - a middleware a la TiTiler but for tiled 3D formats. I'm leaning towards loaders.gl for this discussion as the end goal of the library is to basically support any geospatial format (raster, vector, 3d etc), and be the middleware for any application that want to ingest, display and process such formats - with a strong focus on deckgl/mapbox/maplibre ecosystems.

TiTiler main use-case if to ingest COG (Cloud Optimized Geotiffs) and serve a TMS/XYZ endpoint on the fly - without storing the tiles other than via caching. This way, any application (web or desktop) that do not yet support the tiled/pyramid structure optimization of COG can still access them via a standard, old-school TMS endpoint. It was ignited and is being extended by folks at DevelopmentSeed like vincentsarago and kylebarron - with similar projects like TiFeatures, TiPG, TiMVT etc) on top of FastAPI, can support STAC catalogs, on-the-fly mosaicking, styling etc.

Most softwares from both the geospatial sector and 3D-rendering engines made the shift to spporting 3D-tiled formats

• Qgis loads both COPC (tiled pointclouds) thanks to a dev by hobu and OGC 3D-Tiles via a dev by nyalldawson following groundwork by oslandia
• web apps via loaders.gl, deckgl, support on threejs via NASA-AMMOS/3DTilesRendererJS
• Unreal Engine by Epic Games, Unity, Open3DEngine, Nvidia Omniverse etc all support OGC 3D Tiles via the corresponding plugins developped by the Cesium team
• ESRI I3S etc

Using such a middleware would make any of these apps ingest any 3D-tiled datasets (mesh or pointcloud, COPC, OGC 3D-tiles, potree etc, really anything supported by loadersgl). Translating a request tile index, computing a rigid matrixTransform from one CRS to another CRS on the fly, using the underlying already tiled representation of the input data would probably make it doable - might require to have an underlying unified representation to translate to/from.

Would love to hear anyone's thoughts about this!

[ibgreen]

@jo-chemla loaders.gl has both 3D Tiles and I3S support, plus a two way converter between the formats.

We have discussed on-the-fly conversion between the formats (that is how titiler works, after all) but it is surprisingly difficult. I3S has node page structure and other things that span many tiles and it is not just a 1-1 transform.

Instead loaders has pluggable loaders of the same category that output data in a standardized form (3D Tiles and I3S being in the Tiled 3D category).

Then the deck.gl Tile3DLayer can render that standardized data, making it easy to add support for a new Tiled 3D format (say COPC or Potree).

If you see an area where you could contribute let me know.

[jo-chemla]

Thanks for getting back. That's interesting to know - indeed, each 3d-tile flavor (either OGC 3d-tiles, Esri I3S, Potree etc) has its own hierarchical representation + way to store geometry, material etc.

Good to know the way loaders handle it is by using a standardized representation, where each loader can output to that representation. I imagine the I3S <-> 3DTiles conversion is made possible by the fact that the plugins can not only read (from input to that representation) but also write = convert from that standardized representation to the corresponding data standard.

Having a way to do such requests on the fly would indeed be amazing. A client could ask for a type A: wanted 3D-tile root tileset and tile content at specific indices, and the middleware would convert that request to the intermediate type B: standardized format, and finally from the standardized representation to the type C: dataset standard. Or that middleware would rather fetch any data necessary from the input tiled dataset, and build up the response formatted as the requested format.

Would love to know if these concepts could fit and be integrated into loadersgl, and if so where would be the best place to start - and whether this type of effort would be something of interest for other users, especially data-producers (3d-scanning companies, national institutions etc) and aggregators (gis, cad, 3d rendering engine users and devs).

[ibgreen]

There are different ways to approach this, so if you want to start coding then it is just a matter of picking the approach you are most excited about / where you are likely to be most productive.

• Adding COPC and/or POtree support.gl would be a concrete approach. You should then be able to make a 3D example with a drop down that lets the user pick any URL and it will be displayed by the same minimal app using the deck.gl 3D tile layer.

• You could also make a composite deck.gl layer that renders different actual layers TIle3DLayer, TIleLayer, GeojsonLayer etc) depending on the data loaded by loaders.gl and this way you would have the "everything layer" you are envisioning - just throw data or URLs at it and it will figure out how to load and render them.

• I currently don't see too much use for on-the-fly conversion as we already implemented support for both formats, especially given how much work it is to implement on-the-fly conversion. Perhaps it could bring I3S/COPC/Potree support to other JS frameworks that only have 3D Tiles support.