Vector tiles support

[yohanboniface]

For what I understand now, we have those subtasks:

• add support for consuming vector data sources => core
• add support for opening a tm2 style folder => core or plugin (Loader)
• add support for serving local data source ('tmsource://') => plugin
• tilejson support
• metatile support
• local vector tile tree directory support
• add support for MapBox oauth to be able to consume their vector tiles API => optional, plugin
• image export

Though, many areas of the process remain confused for me at this point.

cc @pnorman in case you want to 1. add some elements 2. track the progress of the issue

[yohanboniface]

Here is an example of the minimal workflow for merging a vector source and a style to produce an image tile:

var mapnik = require('mapnik'),
    fs = require('fs');
var vtile = new mapnik.VectorTile(0, 0, 0);
vtile.setData(fs.readFileSync('./test_vector_tile.pbf')); // Comes from http http://vector.mapzen.com/osm/earth,landuse,roads/0/0/0.mapbox
vtile.parse();
// console.log(vtile.names());
// console.log(JSON.stringify(vtile.toGeoJSON(0)));
var map = new mapnik.Map(vtile.width(),vtile.height());
map.fromStringSync('<?xml version="1.0" encoding="utf-8"?>'+
'<!DOCTYPE Map[]>'+
'<Map srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over" background-color="steelblue">'+
'<Style name="style" filter-mode="first">'+
'  <Rule>'+
'    <PolygonSymbolizer fill="#efefef" />'+
'  </Rule>'+
'</Style>'+
'<Layer name="earth"'+
'  srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over">'+
'    <StyleName>style</StyleName>  </Layer>'+
'</Map>'+
'');
map.extent = [-20037508.34, -20037508.34, 20037508.34, 20037508.34];
vtile.render(map, new mapnik.Image(vtile.width(),vtile.height()), function(err, vtile_image) {
    if (err) throw err;
    vtile_image.save('test_vector_tile.png', 'png32');
});

One of the traps is that, we need to have those "dummy" layers (without Datasource) in the style, so the one who creates the style Mapnik XML needs not to know the source layer to be able to create it.

So, basically, as I see it now, kosmtik would:

• parse the project.yml
• taking care of the styles if any attributes to generates the Stylesheet property
• taking care of the layers one if any to generate those "dummy" Layer
• when generating a tile image, if a source attribute is present, we fetch the vtile from the source, and generate the image from here
• certainly a cache will be need (to be totally cleared on each project reload: when running kosmtik and each time a project file is saved)

Question: should we try to follow the exact project structure and convention than a tm2 project, or should we take the liberty of some custom choices when it makes sense AND having a "tm2 compatibility layer". Option 2 means a bit of duplicate work, but in the same time I'm not sure we want to be totally reliant of tm2 choices that can change anytime in the future, so it may makes sense to promote a more generic/open vector tile project structure.
Certainly I will go for option 1 as first choice in short term, but the question need to be discussed.
Just an example: in MapBox Studio, a source without protocol means a MapBox Id, AFAIK, this is not a fallback that makes sense outside of MapBox products.

[pnorman]

MapQuest/avecado#31 (comment) might be relevant.

Couple of things I've ran across

• tm2source:///full/path/to/source.tm2source is equivalent to appending data.xml on the end and rendering vector tiles from that Mapnik XML
• Sources are URI and need to have a protocol unless they're relative, which doesn't make sense in this context
• There's no good way to express relative locations with local files, which makes a multiuser workflow a pain for something like https://github.com/MapQuest/mq-carto-style-vector/blob/master/mq-carto-style.tm2/project.yml#L22
• If you point to a remote source (e.g. http) are you pointing to the tilejson for it, to the root (portion pre z/y/x.pbf), or to something else? I think TM2 and Avecado both take the tilejson view.
• For pointing at local files, how do you distinguish between pointing at an XML file and rendering, or pointing at a set of pre-rendered vector tiles?

[yohanboniface]

tm2source:///full/path/to/source.tm2source is equivalent to appending data.xml on the end and rendering vector tiles from that Mapnik XML

Good point; I think a kosmtik plugin (or core?) could serve that, so we have a full worflow support

Sources are URI and need to have a protocol unless they're relative, which doesn't make sense in this context

But what about mapbox://a-mapbox-id?
I'm thinking about exposing to plugins the ability to register some hook per protocol, so we don't have to add MB specific cases in the core, but we can have a plugin for people that specifically want to have tm2<=>kosmtik compat

There's no good way to express relative locations with local files, which makes a multiuser workflow a pain for something like https://github.com/MapQuest/mq-carto-style-vector/blob/master/mq-carto-style.tm2/project.yml#L22

Even if it's not the definitive answer for this, the "local config" feature of kosmtik can be of help here. See examples of my localconfig for HOT rendering (both .json and .js examples here, but only one should be used in the same time, indeed).

If you point to a remote source (e.g. http) are you pointing to the tilejson for it, to the root (portion pre z/y/x.pbf), or to something else? I think TM2 and Avecado both take the tilejson view.

Good question. I think tilejson is a great standard answer for this, but in the same time being able to simply support the {z}/{x}/{y} de facto standard tile scheme should be good too, to keep things simple. Maybe a protocol can help? zxy://?
Also, good to note that MapZen tiles URL doesn't end with 'z/y/x.pbf' or 'z/y/x.vector.pbf', but with 'z/y/x.mapbox' (when we want pbf). So we can't make the URL path implicit.

For pointing at local files, how do you distinguish between pointing at an XML file and rendering, or pointing at a set of pre-rendered vector tiles?

Humm, another good question. Maybe we can sniff the presence of "project.xml"? Or we can rely on the 'xxx.tm2source' convention about the folder naming when it's a source to be served, and otherwise it's a tree of tiles?

[yohanboniface]

About sources URI.

We can have:

• a path to a local tm2 source data dir
• a path to a local mapnik xml
• a remote URL to a tilejson that gives the TMS URL template to use
• a TMS template
• [non standard] a Mapbox id

So here is a first suggestion of the URLs formats to be parsed:

• tmsource:// => a local tm2 source that need to be served
• file:///xxxx.xml => a mapnik XML file to serve vector tiles from
• dir:///xxxx => a tree of pregenerated vector tiles
• http(s):// => URL of a tilejson
• tms:// => a TMS template

Not sure about what to do when protocol is missing (tm2 project makes implicit that it's a Mapbox id, AFAIK).
Also, the OAuth question will need to be addressed at some point.

That's a first shot of reflection, and I'm certainly missing some points.

[pnorman]

Given that there's no reason a relative path can't be changed to a mapbox:// URI, I don't see a need to process a relative URI.

[yohanboniface]

Pushed a first poc, but there is still work to do:

• tilejson support: I need a open example to work with
• metatile support: not sure about how to handle the metatile construction with vectortile, given that AFAIK Mapnik vector tiles only work on 256x256 for now

[pnorman]

• metatile support: not sure about how to handle the metatile construction with vectortile, given that AFAIK Mapnik vector tiles only work on 256x256 for now

There's two reasons for metatiles - efficiency and labeling.

With rendering direct from database, you take a certain amount of time that is essentially independent of area rendered to go through the indexes and do at least one seek on the table. Thus, you want to render enough features at once for efficiency reasons. A 8x8 metatile probably takes about as long as 4 individual tiles.

With vector tiles, you don't have that, and the only static costs is a round trip to initiate tile retrieval and setup for rendering an area. Everything else should be closer to proportional to number of features, meaning a 8x8 metatile should take closer to the time of 64 tiles.

This isn't to say that there wouldn't be some throughput increase for metatiling.

Labeling is... complicated. Off-hand, I'm not sure how you merge adjacent vector tiles and handle their buffer overlap.

I guess what I'm saying is, don't worry about metatiles.

[pnorman]

tilejson support: I need a open example to work with

I'll see what I can do. Or you could generate one for https://github.com/mapzen/vector-datasource/wiki/Mapzen-Vector-Tile-Service

[yohanboniface]

Or you could generate one for https://github.com/mapzen/vector-datasource/wiki/Mapzen-Vector-Tile-Service

I've opened an issue for that, they seem to this: tilezen/vector-datasource#73 (I could work on a PR there but I guess I've enough to do on the kosmtik side)

Labeling is... complicated. Off-hand, I'm not sure how you merge adjacent vector tiles and handle their buffer overlap.

One thing I've in mind is, for a given metatile, to call all the vectortiles, and then to call setData for each. Not sure at all it'd work, but I want to try. Also I wonder if the "overzooming" thing is not part of the answer.

Not having metatile support means that we don't have the final labelling behind the eyes while working, which is not what we want indeed. Another problem is for the exports like tiles tree folder or MBTiles (I'm about to work on this, because I will need it soon).

[pnorman]

One thing I've in mind is, for a given metatile, to call all the vectortiles, and then to call setData for each. Not sure at all it'd work, but I want to try.

There's all sorts of stuff that won't work on a conceptual level, such as ordering.

A couple are

1. If tile 1 has a layer with features A, B and tile 2 has a layer with features C, D, how do you join them?
2. Interior buffers are non-obvious. If you cut them off, this will lead to errors with some line cap types that wouldn't have been there before. If you leave them there, then you'll get overlap. The former would be most obvious with a butt line cap and a line that changes direction sharply at the tile edge and the latter most obvious with a transparent line which then overlaps.
3. You can bet that this would screw up labeling somehow. Particularly at edges.
4. To avoid loss of precision, the Layer extent would need to be altered from the default. This is probably not well tested.

Not having metatile support means that we don't have the final labelling behind the eyes while working, which is not what we want indeed

The server stacks I've seen do tile by tile rendering, so labeling is the same.

Another problem is for the exports like tiles tree folder or MBTiles (I'm about to work on this, because I will need it soon)

I don't see why MBTiles would be an issue. You generate tile by tile and stuff them into the sqlite database.

Where we'd see bigger MT gains would be MTs on generation of the vector tiles, but it's not clear there if it's worth it for continually updating data. Profiling would be required.

[yohanboniface]

I don't see why MBTiles would be an issue. You generate tile by tile and stuff them into the sqlite
database.

It's not about the MBTiles itself. It's the fact that we can tolerate to have non optimized label placement when coding, but when we export the map (tree dir or MBTiles) to be used in some production situation, it becomes a problem.

[pnorman]

We should have the ability to re-order layers within the resulting Mapnik XML. Rendering order is based on that order, not the order within the vector tile.

[yohanboniface]

For now, is adding a "layers" key in you style yml an option?

layers:
  onelayer
  anotherlayer

Still in the style yml, you can also just add the layers name as a list in the Layer key, they will be normalized.

Any other suggestion?

[pnorman]

I don't actually need the ability to re-order layers for my current work since I'm in control of the vector source

[bcamper]

Just wanted to note that TileJSON endpoints are now available for the Mapzen Vector Tile service (http://mapzen.com/vector). See tilezen/vector-datasource#73.

[yohanboniface]

Excellent! Thanks for that, I'll test that very soon :)

[denics]

This issue is very old, what is the status of vector tiles support in kosmtik?