[FEATURE] Vector tile layer - part 1

[wonder-sk]

This is the initial work on vector tile layer support 🎉

The implementation is based on what has been previously discussed in Vector Tiles QEP:
qgis/QGIS-Enhancement-Proposals#162

(The red lines are not rendering artifacts, they are drawn intentionally to show tile borders.)

Summary

So far this only includes changes to qgis_core library.

Main additions to the public API:

• QgsVectorTileLayer - the main map layer class
• QgsVectorTileRenderer - base class for renderer class implementations for vector tiles
• QgsVectorTileBasicRenderer - default renderer implementation

New private classes (may be added to public API if needed):

• QgsVectorTileLoader - handles fetching of tiles from network / mbtiles
• QgsVectorTileMVTDecoder - handles decoding of raw tile data to features
• QgsVectorTileLayerRenderer - takes care of overall rendering (fetch + decode + draw)
• QgsVectorTileUtils - various useful functions

Some new classes that may be shared by vector and raster tiles:

• QgsTileXYZ - position of a tile in tile matrix set (zoom level, column, row)
• QgsTileRange - rectangular selection in a tile matrix (start/end column and row)
• QgsTileMatrix - description of a tile matrix (map extent, zoom level, scale, number of rows/columns)

(also QgsTileMatrixSet to be added later when we support other tile matrix sets than just GoogleCRS84Quad)

Testing

You can load a vector tile layer from Python console, e.g.:

ds = QgsDataSourceUri()
ds.setParam("type","xyz")
ds.setParam("url", "https://api.maptiler.com/tiles/v3/{z}/{x}/{y}.pbf?key=YOUR_FREE_API_KEY")
uri = bytes(ds.encodedUri()).decode('utf-8')
vtl = QgsVectorTileLayer(uri, "Vector Tiles Test")
QgsProject.instance().addMapLayer(vtl)

It is also possible to load vector tiles from a local MBTiles file - for "type" use "mbtiles" and for "url" use a local path (e.g. "/home/martin/x.mbtiles").

Rendering / Styling

We will probably need to have a discussion how the default renderer should work (I call it "basic"). It is currently based on rules ("styles") that refer to sub-layer name, geometry type and filter expression, and they have a QgsSymbol assigned. This is similar to what other rendering engines support. In theory in QGIS we could do more and support any vector renderer (categorized/graduated/rule-based/...) but for now I have kept things simple.

The other consideration is how to deal with existing styles - at this point most existing styles are done using Mapbox GL JSON style format. It is a question whether we should have a custom tile renderer which would try to emulate the native rendering as much as possible -- or to just have an importer that would convert those styles to our "basic" renderer (likely having somewhat lower fidelity). I have first started with a custom renderer, but over time I am thinking that using our "basic" renderer everywhere would be better also for the future.

Next Steps

Once this PR is merged, there will be more PRs coming to integrate vector tile layers into QGIS application: browser integration, styling GUI, identify tool support, attribute table.

Also there is currently no support for labeling, which should be added as well.

Thanks

Huge thanks to all funders who have contributed to the crowdfunding and made this possible 👏 👏 👏
https://www.lutraconsulting.co.uk/blog/2020/04/02/vectortiles-donors/

[nirvn]

@wonder-sk , this is wonderful! I have a fair amount of thoughts on the styling front, I'll write more tomorrow. But the general line being, IMHO we need to be trying to match tile styling as much as we can with that of non tiled vector layers. And we need to make sure some of the basic QGIS environment expectations, such as feature based data defined properties, are met.

Great work! And thanks to the funders

[nyalldawson]

Note that if you're after maximum speed, the fastest like string constructor is the one which accepts seperate double vectors for X and y

[wonder-sk]

Good point... but from some timing test I did the decoding phase is quite fast - it is actually evaluation of filter expressions that's the slowest bit. If you don't mind I would leave this unchanged. (+ I find that separation of vectors for X and Y in QgsLineString is very strange - it should be more cache friendly if they were together, and just one array instead of two.)

[nyalldawson]

Mmm possibly? But it'd need to be an array of doubles and not QgsPoint (don't want to store all those unused z/m values). And then you'd need to upgrade all the existing performance sensitive code which uses the two vector constructor... 😜

[wonder-sk]

Yeah an array of doubles is what I had in mind too... and yes, it would be quite some work to get all that converted... maybe one day :-)

[nyalldawson]

Will there be data providers for vector tiles?

[nyalldawson]

The red lines are not rendering artifacts, they are drawn intentionally to show tile borders.

I'm curious to know if there any visible artefacts under these lines - does the antialiasing of the clip rectangles cause visible edge effects?

[nyalldawson]

I have first started with a custom renderer, but over time I am thinking that using our "basic" renderer everywhere would be better also for the future

I'd also tend towards conversion vs custom renderer

[wonder-sk]

Thanks for the review @nyalldawson

Regarding colors - I was unsure how set up default renderer. In theory we would want to show each sub-layer with a different color, but given that we may not even know all possible sub-layer names, I thought it would be best to use the same color for all layers. For the particular choice of default colors - I will be happy with any better colors people suggest :-)

Will there be data providers for vector tiles?

I did not need the concept of data providers so far, so I have not introduced them... It would make most sense to create data providers when we know what kind of abstraction do we need.

I'm curious to know if there any visible artefacts under these lines - does the antialiasing of the clip rectangles cause visible edge effects?

I have not seen any artifacts so far! I was worried about that initially, but it seems that Qt is doing very good job with clipping (and I tried it also with reprojection when tile boundaries were not aligned with axes).

[nyalldawson]

Regarding colors - I was unsure how set up default renderer.

Could you call QgsColorSchemeRegistry::fetchRandomStyleColor() ? that should give one not-random-but-pretend-random (nice) color per call.

For the particular choice of default colors - I will be happy with any better colors people suggest :-)

Failing all else, I'd pick the first color from: QgsColorSchemeRegistry::randomStyleColorScheme()

I have not seen any artifacts so far! I was worried about that initially, but it seems that Qt is doing very good job with clipping

Great to hear! I was concerned given the long-standing issue/limitation regarding neighbouring polygon bounds showing through due to antialiasing of the edges. (Maybe clipping paths completely skip the antialiasing on the boundary?)

[wonder-sk]

I think this PR should be ready for merging...

[nyalldawson]

Wouldn't this be better as a render context flag?

[wonder-sk]

So it would be a global flag applied to all layers? And probably to raster tiles as well?
I don't know... I would probably prefer to keep it as is, maybe it can be moved to render context later. There are few more things like having tile XYZ numbers printed or border line/color configuration that would be nice to have too, but not sure if they all belong to render context.

[nyalldawson]

Yeah, it just struck me that it applies to other layers/providers too so would be nice to have something less ad-hoc to handle it. Could we hide from sip at least so it's not stable API?

[wonder-sk]

Hmm... removing it from sip would kind of remove the whole point of having it in API at all - I wanted to be able to turn this flag on/off from python console. I would be also fine with removal of this flag before the release if you don't like it, but now it is helping me with development... :-)

As a side note - it would be probably reasonable to mark vector tile layer related stuff as experimental API for now and pronounce it stable e.g. for 3.16 ...?

[nyalldawson]

+1 to merge after above comments are resolved

[nyalldawson]

Oh, you'll also need to get the azure builds fixed before merging!

[saberraz]

Hmmmm...I now get this error when I try to compile (even from a fresh build dir):
'../src/core/protobuf::protoc', needed by 'src/core/vector_tile.pb.cc', missing and no known rule to make it

[wonder-sk]

This needs protobuf in osgeo4w to be compiled as a dynamic library. Right now it is built as a static library and the build fails on Windows...