Host vector tiles generated by tilemaker

[pnorman]

The OWG is planning to host vector tiles generated by tilemaker as an experiment. The goal is to get something others can work with to see where more work needs to be done

The intended architecture is

• weekly downloads of the planet file and generation with tilemaker (est. 2-3 day runtime) to create a mbtiles artifact
• running a mbtiles server, perhaps https://github.com/systemed/tilemaker/blob/master/server/server.rb#L30 behind apache
• potentially offering the raw mbtiles for download
• reusing the IP rate-limiting code for 429s
• using fastly as a CDN

We have some resources we can potentially reuse for this, but the immediate priority is scaling up tile.osm.org with the new dublin and umea servers.

Items that need addressing are

• How much space does a mbtiles file for the planet need? @systemed
• How much RAM does tilemaker need for the planet? @systemed
• Is there a way to reuse parts of another Fastly config to avoid duplication, particularly of useragent blocks

[imagico]

I know you indicate this to be an experiment but still - may i ask what the strategic vision is behind this or in other words: Where do you intend this to go if the experiment should be successful (by whatever definition of successful)?

I mean is it supposed to lead to the OSMF providing a vector tile service for other parties (and if yes: how does it intend to position that relative to commercial services in that domain)? Or is it supposed to be a non-monetary support program/grant for tilemaker development (and if yes: on what basis was it decided to choose tilemaker to receive that support and not other software in the domain of map rendering/data preprocessing for map rendering)?

As you know i have been saying for quite some time that the OSMF needs to strategically invest in development of technologies suitable for scalable high quality cartography based on OSM data and usable for cooperative community projects so in principle i would see favorably any steps in that direction. But i would like to understand how this step fits into such a strategy or if not what other strategic goals it is meant to serve.

[tomhughes]

The immediate goal is to support the development of vector tiles for openstreetmap.org.

[imagico]

Trying to understand what you mean by that - so this is a feasibility study to see if tilemaker can be practically managed operationally to generate world wide vector tiles (with no requirements what that entails specified so far, presumably within the current capabilities of tilemaker) with a weekly update cycle and then recruit some volunteer style developers to develop a map style (or several - presumably using Maputnik) to produce maps targeting yet to be specified needs on openstreetmap.org.

I see a number of issues with that (in particular in the long term regarding the strategic needs for community map styles providing meaningful constructive feedback for mappers). But this is evidently not a very suitable venue to discuss such matters (although i would very much suggest you to have such a discussion with a broader reach - not many people follow this tracker and few would be inclined to comment here).

Independent of the specifics discussed here i would suggest to clearly distinguish between operational work (stuff that serves concrete and well specified immediate needs) and strategic endeavors and programs. When you state the goal to support the development of vector tiles for openstreetmap.org it is not clear if that is an operational goal (in which case it seems a bit underspecified - see above) or strategic endeavor (in which case a more long term vision what strategic goals this should serve would be highly advisable IMO).

[tomhughes]

What I said on the operations call yesterday (and this is my personal view) is that I hoped that providing them would break the impasse that seems to exist where people can't work on a style because there's no tiles but don't work on tiles because there's no style.

My hope would be that people will start developing a vector style we can use and that there can then be a feedback cycle between that and any necessary changes to the tile schema until we have something we feel is deployable.

At that point we can worry about what stack to use in production and how best to generate and serve vector tiles for that.

[imagico]

Understood. Just keep in mind that when the OSMF rolls out something like that there will be a message of endorsement (of the tools used, their capabilities and their limitations and the standards they are built upon) and a message of strategic direction (in what direction community cartography in the OSM community should develop) perceived even if no such message is intended.

[bdon]

(edit: not relevant to this ticket)

This is exciting! I'm new to this effort, but I'm wondering it the intent here is to offer an experimental vector layer visible on OSM.org? I predict needing some messaging to manage expectations around it not being an alternative or successor "basemap" to OSM Carto (even if that is a long term goal, it would depend on heavy development of a cartographic product like Tom described)

What would be immediately useful is if the vector layer provided both:
1) a language switcher to change the name: tag rendered by by the browser
2) integration with OSM.org User Preferences language settings to default to those language tags

Even if the vector layer displays only place= points+relations and no other geometries, it would be immediately useful as a tile-based web visualization of multilingual data already in OSM; OSM Carto only shows name, and iD only works at very high zoom levels. Even better would be if the rendered name tags were clickable links to a /node/12345/edit page.

[tomhughes]

Please can we keep this ticket on topic and not go off on crazy tangents.

This ticket is just about getting an experimental vector tile source deployed.

Writing a vector style is a matter for other people to be discussed in other places.

Deploying that style to openstreetmap.org is a matter for operations and/or the web site developers once it exists, which is a logn way off.

Once we have the tiles we will probably put some sort of minimal style up on a separate domain as a way of visualising them in the short term, with the hope that somebody will then start to develop a full scale style that we can then put in it's place until it's ready to consider a production deployment.

We are all well aware of the additional opportunities that vector tiles present, which is why we are trying to encourage people to work in that direction, but this is not the place to discuss wider issues.

[nyurik]

Tilemaker explicitly states:

But don't use tilemaker if:

• You want the entire planet

Has this concern been addressed, or is there a workaround of sorts that might not be good for others? While we could say that the stack discussion should be postponed, the specific settings to convert download into tiles is stack-specific, so deploying a solution that is not optimal for others could in theory lead to duplicated efforts.

P.S. This is a great news regardless, thank you for working on getting MVT to OSM!

[tomhughes]

We literally had the author of tilemaker on the operations call on Wednesday to discuss this so yes, we have discussed it.

[Firefishy]

RAM estimate requirement is between 100GB and 200GB. Estimate time to generate for the full planet is around 4 days runtime on a 2x Intel Xeon X5650 using 144GB RAM system.

@systemed would be able to provide more clarity.

[systemed]

Yes, that's about right; hopefully a bit less time than that. Currently finishing off a set of performance optimisations which will give some harder numbers.

[klokan]

If I understand it well, the goal of the experimental vector tile service is to boost the OSM vector tile style development and to improve the schema for OSM vector tiles - and allow regular iteration on development by the community.

The community around @openmaptiles would be very happy to help with these efforts.

For the upcoming OMT v4 these things may be of interest to OSMF:

• Near real-time updates and serving are technically possible
• An “OSM Carto” type of style - with cartographical goal to demonstrate all included data - developed together with the schema improvements (ideally community PRs to contain both modification of the style snippet and OSM tags / zoom / generalization selection)
• CI tests on every contributed PR - to validate size&speed increase
• Maxzoom 15 in VT schema considered, probably required for a style like OSM Carto, to avoid huge z14 tiles in some places
• Ability to enrich/redefine layers - with different maintainers for separate layers
• All features Include OSM IDs to interact with OSM endpoints - a la https://osmapp.org/

If OSMF is interested - we could provide a weekly updated downloadable tile package for self-hosting and/or live tile endpoint (for proxy via your CDN or direct use on your DNS), based on the latest OSM data and powered by the latest modifications of the schema in GitHub. You could also run the open-source components on-prem with Kubernetes. Just let us know.

Delay on pre-generating an entire planet is about a day - less if materializing only upper zoom levels, with real time on lower zoom levels served from an endpoint, or if only diffs are generated.

[pnorman]

Hosting tilemaker as an experiment does not preclude us from hosting other vector tile solutions. This is something we're planning on hosting using resources that will become available later this year. It will not be a featured tile layer on the website to start, and if it should be is a question the OWG will consider at the time. We have in the past hosted experiments that did not end up on osm.org, and looked at multiple approaches for a goal.

For using a live tile endpoint, are you thinking of that on osm.org? We had spoken to someone back in March with the call for featured layers, but communication died out. If you're interested in that, please email information about it and meeting the criteria of the featured layer policy to the OWG.

If you're thinking of instead asking the OWG to host an instance of the OMT stack, please open a new issue so this one remains about maptiler.