Tile serving code roadmap

[nyurik]

As a MapLibre project, we should have a set of components to build tile serving infrastructures with these features/requirements:

• Support multiple data stores (.mbtiles, PostgreSQL, disk files, ...) plus some metadata (tilejson, ...)
• Web service that serves metadata+tiles, static files like the maplibre-gl-js, fonts, TLS, health, logging, ...
• Optional services like backend tile rendering, geojson to tile conversion, ...
• User-provided web route configurations and user services integrating with the above in the same process.

Some users who build standalone apps with maplibre-rs or maplibre-native may need to get data from a local storage rather than web, i.e. offline tile storage. The above datastore/service separation would solve this issue. cc: @maxammann

What's available today?

mbtileserver-rs

MapLibre/mbtileserver-rs is a good small tile service written by @ka7eh that supports .mbtile (SQLite) files and simple web requests (http only). The project is not very well known (21 stars, 5 forks), and it would need some significant work to fulfill all of the above requirements. At the same time, it is the only project I know of this nature with mbtiles support.

Martin

The owners of another extremely popular project Martin (82 forks, 741 stars), written by @stepankuzmin, has expressed desire to join MapLibre and continue working on Martin. This project is fairly up to date with web serving code, and uses PostgreSQL as a data store.

T-Rex

t-rex tile server (49 forks, 450 stars) is another great project by @pka that is similar to Martin but has a slightly wider scope. This project also uses PostgreSQL and GDAL as data sources, has some automatic configuration, built-in viewers for data display and inspection, tile generation, and some other interesting functionality that overlaps with the above requirements. I have not spoken with @pka about collaborating yet, but I'm sure at least some components could be shared with their efforts.

One ring to rule them all

otherwise known as we have 14 standards (XKCD)

In order to satisfy the above goals, I would like to propose a providers model, similar to the tilelive API -- a common interface able to provide tiles by its xyz coordinates.

• refactor Martin's PostgreSQL datastore code into a separate lib, possibly using some of T-Rex code as well.
• refactor mbtileserve-rs .mbtiles code into a separate lib
• enhance maplibre-rs to be able to render tiles accessible via the same interface
• Martin, maplibre-rs, and mbtileserve-rs (?) will use any of the data providers as needed.

See also maplibre/mbtileserver-rs#16 - a discussion about mbtileserver-rs restructuring.

[pka]

There is a severe problem in the current MVT tile serving landscape. There are so many tile servers, that none of them gets enough funding. So it's really a XKCD-like solution to write another one. And that's what I'm planning to do myself as well 😄 My plan is to use goezero as data format abstraction and put an OGC conformant tile server on it. If the maplibre project has excess capacity to go in that space as well, then my advice is to use a mature web serving framework like Actix.

[maxammann]

I think using geozero as the default interface for geodata is very good. But I think we are missing an abstract interface for loading geo data over differnt protocols. Thats what tilelive is/was. It put an abstraction over the transport of geodata.

So both interfaces are equally important imo. For example you want to be able to load MVT through: sqlite/mbtiles, HTTP, S3 etc.
You also want to be able to load GeoJson over the same protocols.

OCG conformance is probably another layer which is quite independant of the data format and protocoL/transport layers.

[brendan-ward]

It is still under active development, but GeoParquet (and associated GeoArrow) is an emerging specification similar in intent to geozero but with potential for broader adoption across the geo ecosystem (don't get me wrong, geozero is a great idea). Support for file I/O landed in GDAL 3.5, so would be available via something like t-rex, but one of the really interesting things happening in that space is exposing an Arrow API directly from within GDAL. This would be in the data source => MVT tile space. (it is also possible and relatively straightforward to consume from these formats without going through GDAL).

However, the way I'm reading this discussion is more about the tile API that sits between the data source and web APIs, so I'm not sure how relevant that is other than to increase awareness of that activity.

[pka]

GeoArrow driver for geozero is in the works: georust/geozero#39

[wipfli]

And forking tilelive is not a good idea I assume?

[wipfli]

And shapefiles?

[nyurik]

tilelive defines two abstractions: an API to get tiles (set of getTile(z,x,y)-like functions), and the API to register/instantiate data providers (register protocol and instance parameters as a URL string). I am all for establishing the first one, but I'm not too certain the second one should be in scope.

[brendan-ward]

I'd suggest that abstracting a tile interface that can broker between arbitrary data sources (mbtiles, postgres, GIS files, etc) for use in web APIs should be a separate concern than providing a vector data abstraction interface between arbitrary data sources and consumers; otherwise you risk growing the scope too much. The tile interface seems relatively narrow in focus and a good place to start re: this discussion. The latter seems like it would draw heavily from the GDAL vector API (OGR), to which I'd suggest that both row level and columnar APIs be given strong consideration in the design; the columnar approach in particular promises to unlock significant performance benefits. Both the tile interface and geo interface could be assembled to realize some of the goals here, but at different levels within the stack.

I'm not entirely sure I'm following the here discussion yet, but I like the idea of defining a generic tile API inspired by Tilelive (esp. getTile, getInfo / getMetadata) - which would then enable us to define a consistent API to expose, e.g., an interface against mbtiles files within a hypothetical mbtiles-rs crate that could be consumed and used within arbitrary web APIs; similar could be done for other data sources. I'm not yet familiar enough with Rust to understand the tradeoffs involved in the callback pattern in the Tilelive API, though I can appreciate that some tile sources will likely need to be async friendly (e.g., if they have to make HTTP requests to data sources in order to assemble the tile).

There are aspects of the Tilelive API that are perhaps a bit outmoded now; support for UTFGrids could be skipped for now unless there are clear use cases.

[nyurik]

I think the interface should only support async - there are very few cases when synchronicity would benefit performance. Do you have an example of a columnar API you have in mind for this usecase? I agree about keeping it to the bare-minimum tilelive-like without grid support.

[brendan-ward]

Columnar API from above: OSGeo/gdal#5830

But this isn't for the tile API, it would be for the reading of arbitrary GIS data API.