New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vector Tiles on osm.org #214

Open
bhousel opened this Issue May 21, 2018 · 41 comments

Comments

Projects
None yet
@bhousel
Member

bhousel commented May 21, 2018

From @bhousel on May 21, 2018 19:49

I've been reviewing the Top 10 Tasks list, and noticed that several of the issues reference vector tiles as a possible way forward.

This is also something that pops up for discussion sometimes here in this repo (#1672 #1623 #1316)

Let's start a discussion about what it would look like to add a vector tile option to openstreetmap.org! Even if it's just a trial or staged rollout available only in some areas, I think it would be great to explore vector tiles as a path forward on some of these issues.

I think we all agree that if something is on the Top 10 Tasks list, it's important to enough to people that it's worth doing. It's time!

Copied from original issue: openstreetmap/openstreetmap-website#1875

@bhousel

This comment has been minimized.

Show comment
Hide comment
@bhousel

bhousel May 21, 2018

Member

From @tomhughes on May 21, 2018 20:3

This isn't (oddly enough) a web site issue, it's an operations issue.

Yet another discussion really isn't going to get us anywhere though - we need somebody to maintain a stylesheet and somebody with relevant expertise to tell me what toolchain to setup. The two are of course likely linked.

Member

bhousel commented May 21, 2018

From @tomhughes on May 21, 2018 20:3

This isn't (oddly enough) a web site issue, it's an operations issue.

Yet another discussion really isn't going to get us anywhere though - we need somebody to maintain a stylesheet and somebody with relevant expertise to tell me what toolchain to setup. The two are of course likely linked.

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes May 21, 2018

Member

To explain some more the first step is to switch the default layer to vector tiles - we absolutely want to do that.

Doing that as I say means making a choice of what toolchain to use, and writing any missing bits though I think that is less likely to be needed these days. Then I can write chef recipes to deploy that toolchain.

Obviously we also a suitable stylesheet.

To be honest although I said this is an operations issue, it doesn't really become that until we have somebody with a plan - then making that plan happen becomes an operations issue and/or a chef issue.

Making a plan is what the top ten repo is for right?

Member

tomhughes commented May 21, 2018

To explain some more the first step is to switch the default layer to vector tiles - we absolutely want to do that.

Doing that as I say means making a choice of what toolchain to use, and writing any missing bits though I think that is less likely to be needed these days. Then I can write chef recipes to deploy that toolchain.

Obviously we also a suitable stylesheet.

To be honest although I said this is an operations issue, it doesn't really become that until we have somebody with a plan - then making that plan happen becomes an operations issue and/or a chef issue.

Making a plan is what the top ten repo is for right?

@bhousel

This comment has been minimized.

Show comment
Hide comment
@bhousel

bhousel May 21, 2018

Member

Doing that as I say means making a choice of what toolchain to use, and writing any missing bits though I think that is less likely to be needed these days. Then I can write chef recipes to deploy that toolchain. Obviously we also a suitable stylesheet.

Yes - we discussed a bunch today and it sounds like there are several good options of vector tile stacks to choose from.. Let's use this ticket to compare options and make a decision.

Making a plan is what the top ten repo is for right?

The Top 10 Tasks can definitely steer us in the right direction. But I wouldn't want people to start working on one of these tasks (like Clickable POIs or Multilingual Maps) and get stuck or frustrated.

I'm definitely not the person to evaluate the options, but lets tag anyone with experience in this.
(off the top of my head, @gravitystorm @pnorman @systemed )

Member

bhousel commented May 21, 2018

Doing that as I say means making a choice of what toolchain to use, and writing any missing bits though I think that is less likely to be needed these days. Then I can write chef recipes to deploy that toolchain. Obviously we also a suitable stylesheet.

Yes - we discussed a bunch today and it sounds like there are several good options of vector tile stacks to choose from.. Let's use this ticket to compare options and make a decision.

Making a plan is what the top ten repo is for right?

The Top 10 Tasks can definitely steer us in the right direction. But I wouldn't want people to start working on one of these tasks (like Clickable POIs or Multilingual Maps) and get stuck or frustrated.

I'm definitely not the person to evaluate the options, but lets tag anyone with experience in this.
(off the top of my head, @gravitystorm @pnorman @systemed )

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes May 21, 2018

Member

Well I suspect this is a big enough project that it will need a repo of it's own where a solution can be developed.

I suspect @gravitystorm will want to stay out of this due to his conflicts of interest. We've mostly been pinning our hopes on @pnorman for this one ;-)

Member

tomhughes commented May 21, 2018

Well I suspect this is a big enough project that it will need a repo of it's own where a solution can be developed.

I suspect @gravitystorm will want to stay out of this due to his conflicts of interest. We've mostly been pinning our hopes on @pnorman for this one ;-)

@mmd-osm

This comment has been minimized.

Show comment
Hide comment
@mmd-osm

mmd-osm May 21, 2018

Lots of discussion on this topic already going on here: gravitystorm/openstreetmap-carto#3201

mmd-osm commented May 21, 2018

Lots of discussion on this topic already going on here: gravitystorm/openstreetmap-carto#3201

@pnorman

This comment has been minimized.

Show comment
Hide comment
@pnorman

pnorman May 22, 2018

Collaborator

2/4 of the layers on osm.org are vector tile based - both OpenCycleMap and Transport are, like all of @gravitystorm's stuff is.

There's been some very preliminary discussion at the WMF about asking adding the WMF map style as another featured tile layer, and it's also vector tile based.

@rory has done some work on porting osm-carto to vector tiles, which has the advantage of being able to easily render similar maps without big performance impacts, and the disadvantage of a much more difficult to understand stack, more components, and worse labeling. The only advantages to it for osm.org would be multiple resolution support and lower resource usage for print.

This doesn't become an operations issue until one of two things happens

  1. Someone asks the OSMF to host a style. For this to happen, that style needs to exist, and the software stack associated with it needs to be mature and sanely deployable. My professional opinion is that nothing is there yet.
    Auto-correct suggests deplorable, which can be the state of putting some of these in to production.

  2. Someone hosting a layer asks for it to be included as a new featured tile layer. This doesn't require any OSMF resources.

My work on a new style using newer technology is bolder which is a client-side style using tegola for vector tile generation and tangram for client-side rendering.

Collaborator

pnorman commented May 22, 2018

2/4 of the layers on osm.org are vector tile based - both OpenCycleMap and Transport are, like all of @gravitystorm's stuff is.

There's been some very preliminary discussion at the WMF about asking adding the WMF map style as another featured tile layer, and it's also vector tile based.

@rory has done some work on porting osm-carto to vector tiles, which has the advantage of being able to easily render similar maps without big performance impacts, and the disadvantage of a much more difficult to understand stack, more components, and worse labeling. The only advantages to it for osm.org would be multiple resolution support and lower resource usage for print.

This doesn't become an operations issue until one of two things happens

  1. Someone asks the OSMF to host a style. For this to happen, that style needs to exist, and the software stack associated with it needs to be mature and sanely deployable. My professional opinion is that nothing is there yet.
    Auto-correct suggests deplorable, which can be the state of putting some of these in to production.

  2. Someone hosting a layer asks for it to be included as a new featured tile layer. This doesn't require any OSMF resources.

My work on a new style using newer technology is bolder which is a client-side style using tegola for vector tile generation and tangram for client-side rendering.

@bhousel

This comment has been minimized.

Show comment
Hide comment
@bhousel

bhousel May 22, 2018

Member

Nice @pnorman - bolder looks a lot like what we're aiming for, and that's great that Andy's thunderforest styles seem ready to go today.

Do you think we could set up a vector-tile based demonstration site? Even it's something like beta.openstreetmap.org and only renders part of the world, but it would be a place to demonstrate richer POIs and localization that people are looking for.

We'd need to swap in a client side library that replaces Leaflet, and we could use it to render the existing vector tiles from yours / andy's / WMF's styles? Apologies if I am misunderstanding the steps needed..

Member

bhousel commented May 22, 2018

Nice @pnorman - bolder looks a lot like what we're aiming for, and that's great that Andy's thunderforest styles seem ready to go today.

Do you think we could set up a vector-tile based demonstration site? Even it's something like beta.openstreetmap.org and only renders part of the world, but it would be a place to demonstrate richer POIs and localization that people are looking for.

We'd need to swap in a client side library that replaces Leaflet, and we could use it to render the existing vector tiles from yours / andy's / WMF's styles? Apologies if I am misunderstanding the steps needed..

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes May 22, 2018

Member

I think you're completely misunderstanding ;-)

Any solution that allows those kind of additions to be added to osm.org needs to be the default style and almost certainly needs to support server side rendering as well as client side.

We're a long, long way from being able to add those kind of features to osm.org and I really think you're trying to sprint when we're not even crawling yet.

Paul's proposal is certainly one way forward, but it's an extra on top of what I would like to see happening, which is a plan for moving the default OSMF hosted layer to vector tiles.

Member

tomhughes commented May 22, 2018

I think you're completely misunderstanding ;-)

Any solution that allows those kind of additions to be added to osm.org needs to be the default style and almost certainly needs to support server side rendering as well as client side.

We're a long, long way from being able to add those kind of features to osm.org and I really think you're trying to sprint when we're not even crawling yet.

Paul's proposal is certainly one way forward, but it's an extra on top of what I would like to see happening, which is a plan for moving the default OSMF hosted layer to vector tiles.

@zerebubuth

This comment has been minimized.

Show comment
Hide comment
@zerebubuth

zerebubuth May 22, 2018

Collaborator

Any solution that allows those kind of additions to be added to osm.org needs to be the default style

Would it be possible to add a client-rendered (as opposed to server-rendered) vector tile layer as a non-default layer similar to the cycle, transport or humanitarian layers? Perhaps something based on OpenMapTiles or Nextzen? (Although the latter doesn't currently meet the data freshness requirement, it could perhaps be used during style development.)

Having it as a non-default layer would allow the style to be worked on in parallel with any work around hosting the tiles or implementing server-side raster fall-back rendering.

Collaborator

zerebubuth commented May 22, 2018

Any solution that allows those kind of additions to be added to osm.org needs to be the default style

Would it be possible to add a client-rendered (as opposed to server-rendered) vector tile layer as a non-default layer similar to the cycle, transport or humanitarian layers? Perhaps something based on OpenMapTiles or Nextzen? (Although the latter doesn't currently meet the data freshness requirement, it could perhaps be used during style development.)

Having it as a non-default layer would allow the style to be worked on in parallel with any work around hosting the tiles or implementing server-side raster fall-back rendering.

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes May 22, 2018

Member

Is that possible using leaflet? I got the impression from @bhousel that client side rendering would require use of a completely different toolkit?

My other question would be what proportion of our users such a layer would work for. I don't really have any idea of how things currently as far as client support for the necessary technology goes across the range of browsers and platforms our users use?

Member

tomhughes commented May 22, 2018

Is that possible using leaflet? I got the impression from @bhousel that client side rendering would require use of a completely different toolkit?

My other question would be what proportion of our users such a layer would work for. I don't really have any idea of how things currently as far as client support for the necessary technology goes across the range of browsers and platforms our users use?

@zerebubuth

This comment has been minimized.

Show comment
Hide comment
@zerebubuth

zerebubuth May 22, 2018

Collaborator

Is that possible using leaflet?

I don't know about other toolkits, but Tangram integrates with Leaflet by exposing a LeafletLayer which can be used like this:

var map = L.map('map');
var layer = Tangram.leafletLayer({ scene: 'scene.yaml' });
layer.addTo(map);

I'm pretty sure that it doesn't preclude using other (raster or otherwise) layers, although there might be some lag between the layers when dragging around (i.e: markers).

what proportion of our users such a layer would work for

Tangram should work on any browser with WebGL support, and globally that's currently around 93% of users. To begin with, I think the new client-vector layer would be non-default, and would be disabled where WebGL support wasn't available.

Perhaps, before the client-vector layer could be made the default, we'd want to have a raster fall-back rendering server for those 7% (or perhaps even fewer, by the time we make the switch). However, I don't think the lack of a fall-back at the moment should prevent us from adding a non-default client-vector layer, and allowing work to be started on the style for that. We might end up using something completely different, but this looks like something we could make a start with.

Collaborator

zerebubuth commented May 22, 2018

Is that possible using leaflet?

I don't know about other toolkits, but Tangram integrates with Leaflet by exposing a LeafletLayer which can be used like this:

var map = L.map('map');
var layer = Tangram.leafletLayer({ scene: 'scene.yaml' });
layer.addTo(map);

I'm pretty sure that it doesn't preclude using other (raster or otherwise) layers, although there might be some lag between the layers when dragging around (i.e: markers).

what proportion of our users such a layer would work for

Tangram should work on any browser with WebGL support, and globally that's currently around 93% of users. To begin with, I think the new client-vector layer would be non-default, and would be disabled where WebGL support wasn't available.

Perhaps, before the client-vector layer could be made the default, we'd want to have a raster fall-back rendering server for those 7% (or perhaps even fewer, by the time we make the switch). However, I don't think the lack of a fall-back at the moment should prevent us from adding a non-default client-vector layer, and allowing work to be started on the style for that. We might end up using something completely different, but this looks like something we could make a start with.

@pnorman

This comment has been minimized.

Show comment
Hide comment
@pnorman

pnorman May 22, 2018

Collaborator

We'd need to swap in a client side library that replaces Leaflet, and we could use it to render the existing vector tiles from yours / andy's / WMF's styles? Apologies if I am misunderstanding the steps needed..

Vector tiles require no client-side changes. You can't tell that Andy's styles are vector tile based, nor the WMF ones, unless you know exactly what to look for.

Nice @pnorman - bolder looks a lot like what we're aiming for, and that's great that Andy's thunderforest styles seem ready to go today.

They're not just ready to go today, on osm.org they are vector tile powered, and have been for a few years.


Would it be possible to add a client-rendered (as opposed to server-rendered) vector tile layer as a non-default layer similar to the cycle, transport or humanitarian layers? Perhaps something based on OpenMapTiles or Nextzen?

I had urged Mapzen to suggest this for their Cinnibar style, but it never happened for an assortment of reasons, and is now a moot point.

Tangram should work on any browser with WebGL support, and globally that's currently around 93% of users.

Yep, Tangram or Mapbox GL would work for most website visitors, but many tile like iD, JOSM, QGIS, and others users need raster images, not a client-side style.

Having it as a non-default layer would allow the style to be worked on in parallel with any work around hosting the tiles or implementing server-side raster fall-back rendering.

Yes. Adding it first as an additional featured layer is the only sane way I see this happening.


For the two issues linked above, they could both be implemented with either vector tiles, client side rendering, or client side rendering of vector tiles. If you put client-side rendering as a pre-requisite to the two tasks linked above, then you're putting a non-software development task as a blocker for two software development tasks.

Collaborator

pnorman commented May 22, 2018

We'd need to swap in a client side library that replaces Leaflet, and we could use it to render the existing vector tiles from yours / andy's / WMF's styles? Apologies if I am misunderstanding the steps needed..

Vector tiles require no client-side changes. You can't tell that Andy's styles are vector tile based, nor the WMF ones, unless you know exactly what to look for.

Nice @pnorman - bolder looks a lot like what we're aiming for, and that's great that Andy's thunderforest styles seem ready to go today.

They're not just ready to go today, on osm.org they are vector tile powered, and have been for a few years.


Would it be possible to add a client-rendered (as opposed to server-rendered) vector tile layer as a non-default layer similar to the cycle, transport or humanitarian layers? Perhaps something based on OpenMapTiles or Nextzen?

I had urged Mapzen to suggest this for their Cinnibar style, but it never happened for an assortment of reasons, and is now a moot point.

Tangram should work on any browser with WebGL support, and globally that's currently around 93% of users.

Yep, Tangram or Mapbox GL would work for most website visitors, but many tile like iD, JOSM, QGIS, and others users need raster images, not a client-side style.

Having it as a non-default layer would allow the style to be worked on in parallel with any work around hosting the tiles or implementing server-side raster fall-back rendering.

Yes. Adding it first as an additional featured layer is the only sane way I see this happening.


For the two issues linked above, they could both be implemented with either vector tiles, client side rendering, or client side rendering of vector tiles. If you put client-side rendering as a pre-requisite to the two tasks linked above, then you're putting a non-software development task as a blocker for two software development tasks.

@mboeringa

This comment has been minimized.

Show comment
Hide comment
@mboeringa

mboeringa Jul 9, 2018

I think this is something you all need to look at:

https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/mapping/new-osm-vector-basemap/

I guess this may be based of Rory's work and fork of openstreetmap-carto, or at least heavily inspired by it... I knew this was possibly coming because I talked to one of the people involved at NACIS 2017, just didn't know the current status and that they were releasing it now just for the user conference.

Do try it out on a mobile phone as well, to really appreciate the vector tile's benefits.

mboeringa commented Jul 9, 2018

I think this is something you all need to look at:

https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/mapping/new-osm-vector-basemap/

I guess this may be based of Rory's work and fork of openstreetmap-carto, or at least heavily inspired by it... I knew this was possibly coming because I talked to one of the people involved at NACIS 2017, just didn't know the current status and that they were releasing it now just for the user conference.

Do try it out on a mobile phone as well, to really appreciate the vector tile's benefits.

@kocio-pl

This comment has been minimized.

Show comment
Hide comment
@kocio-pl

kocio-pl Jul 9, 2018

Thanks for the info. Wow, on my laptop it's working really fast, much better than I would expect!

kocio-pl commented Jul 9, 2018

Thanks for the info. Wow, on my laptop it's working really fast, much better than I would expect!

@mboeringa

This comment has been minimized.

Show comment
Hide comment
@mboeringa

mboeringa Jul 9, 2018

Thanks for the info. Wow, on my laptop it's working really fast, much better than I would expect!

Yes, my experience as well. And it appears client side rendering.

mboeringa commented Jul 9, 2018

Thanks for the info. Wow, on my laptop it's working really fast, much better than I would expect!

Yes, my experience as well. And it appears client side rendering.

@imagico

This comment has been minimized.

Show comment
Hide comment
@imagico

imagico Jul 10, 2018

Somewhat off-topic here - this would probably not even be suitable as an additional layer for osm.org let alone as something to be hosted on OSMF infrastructure.

But it is a nice demonstration of consequent use of aggressive way_area filtering. And this also means it is the exact opposite of Rory's work which was meant to port osm-carto to a different rendering platform without changing the rendering results in a significant way.

Trying to move the discussion into a more on-topic direction - as i mentioned in

https://lists.openstreetmap.org/pipermail/talk/2018-June/080902.html

it might be a good idea to discuss and formulate a few things that are considered to be requirements for a map being rendered on OSMF infrastructure. And with that i don't mean the 'wishlist' kind of things this issue was opened for, i mean the base requirements. This issue tracker is of course not necessarily the best place to have such a discussion.

Such a document would give developers of projects that might in the future at some point want to qualify for a function on OSMF infrastructure an important guideline. Right now most developers approach things purely from a technical side and requirements regarding cartography or usability/usefulness are often - if at all - only considered as an afterthought which in many cases leads to fairly bad results.

imagico commented Jul 10, 2018

Somewhat off-topic here - this would probably not even be suitable as an additional layer for osm.org let alone as something to be hosted on OSMF infrastructure.

But it is a nice demonstration of consequent use of aggressive way_area filtering. And this also means it is the exact opposite of Rory's work which was meant to port osm-carto to a different rendering platform without changing the rendering results in a significant way.

Trying to move the discussion into a more on-topic direction - as i mentioned in

https://lists.openstreetmap.org/pipermail/talk/2018-June/080902.html

it might be a good idea to discuss and formulate a few things that are considered to be requirements for a map being rendered on OSMF infrastructure. And with that i don't mean the 'wishlist' kind of things this issue was opened for, i mean the base requirements. This issue tracker is of course not necessarily the best place to have such a discussion.

Such a document would give developers of projects that might in the future at some point want to qualify for a function on OSMF infrastructure an important guideline. Right now most developers approach things purely from a technical side and requirements regarding cartography or usability/usefulness are often - if at all - only considered as an afterthought which in many cases leads to fairly bad results.

@mboeringa

This comment has been minimized.

Show comment
Hide comment
@mboeringa

mboeringa Jul 10, 2018

But it is a nice demonstration of consequent use of aggressive way_area filtering. And this also means it is the exact opposite of Rory's work which was meant to port osm-carto to a different rendering platform without changing the rendering results in a significant way.

I guess that is the almost unavoidably compromise needed to have client side vector tiles work at medium to low zoom for a dataset like OSM, so if you don't rasterize them and output as raster tiles. Moving potentially millions of landuse and water features to a client at e.g. Z8, even if heavily generalized, is probably unfeasible. At those scales raster tiles excel, as you only need to send a few raster tiles.

I also realised the website's opening remark states the use of ArcGIS Pro. I intially thought this might be some (direct) conversion to JSON of the existing openstreetmap-carto style, and ESRI using their own JSON style editor to potentially fine tune and modify the result. But the use of ArcGIS Pro may actually mean this result was achieved in a way much closer to ESRI's own raster tile basemap services, with styling in ArcGIS and use of ArcGIS layer files, and then output a map document to vector tiles from within ArcGIS.

When I talked to someone from ESRI at NACIS last year, we didn't go into the details of this.

mboeringa commented Jul 10, 2018

But it is a nice demonstration of consequent use of aggressive way_area filtering. And this also means it is the exact opposite of Rory's work which was meant to port osm-carto to a different rendering platform without changing the rendering results in a significant way.

I guess that is the almost unavoidably compromise needed to have client side vector tiles work at medium to low zoom for a dataset like OSM, so if you don't rasterize them and output as raster tiles. Moving potentially millions of landuse and water features to a client at e.g. Z8, even if heavily generalized, is probably unfeasible. At those scales raster tiles excel, as you only need to send a few raster tiles.

I also realised the website's opening remark states the use of ArcGIS Pro. I intially thought this might be some (direct) conversion to JSON of the existing openstreetmap-carto style, and ESRI using their own JSON style editor to potentially fine tune and modify the result. But the use of ArcGIS Pro may actually mean this result was achieved in a way much closer to ESRI's own raster tile basemap services, with styling in ArcGIS and use of ArcGIS layer files, and then output a map document to vector tiles from within ArcGIS.

When I talked to someone from ESRI at NACIS last year, we didn't go into the details of this.

@imagico

This comment has been minimized.

Show comment
Hide comment
@imagico

imagico Jul 10, 2018

I guess that is the almost unavoidably compromise

No, it is not (neither unavoidable nor a compromise). But before i get into a rant - this is now definitely off-topic here so i will leave it at that.

imagico commented Jul 10, 2018

I guess that is the almost unavoidably compromise

No, it is not (neither unavoidable nor a compromise). But before i get into a rant - this is now definitely off-topic here so i will leave it at that.

@bhousel

This comment has been minimized.

Show comment
Hide comment
@bhousel

bhousel Jul 10, 2018

Member

To return back to the topic - what would it take to make this the default style on openstreetmap.org? I think the Esri team has done a great job!

Member

bhousel commented Jul 10, 2018

To return back to the topic - what would it take to make this the default style on openstreetmap.org? I think the Esri team has done a great job!

@mmd-osm

This comment has been minimized.

Show comment
Hide comment
@mmd-osm

mmd-osm Jul 10, 2018

what would it take to make this the default style on openstreetmap.org

Isn't this all an ESRI proprietary, closed source rendering stack? If so, there basically no way to run this on osmf hardware I'd say.

Do try it out on a mobile phone as well, to really appreciate the vector tile's benefits.

Unfortuantely, this consistently crashes on my phone after a few seconds, tried Safari and Chrome.

mmd-osm commented Jul 10, 2018

what would it take to make this the default style on openstreetmap.org

Isn't this all an ESRI proprietary, closed source rendering stack? If so, there basically no way to run this on osmf hardware I'd say.

Do try it out on a mobile phone as well, to really appreciate the vector tile's benefits.

Unfortuantely, this consistently crashes on my phone after a few seconds, tried Safari and Chrome.

@bhousel

This comment has been minimized.

Show comment
Hide comment
@bhousel

bhousel Jul 10, 2018

Member

Isn't this all an ESRI proprietary, closed source rendering stack? If so, there basically no way to run this on osmf hardware I'd say.

Ok, so then - Esri open sourcing the tools and style is what it would take? I mean, that's a good place to at least start a conversation.

Member

bhousel commented Jul 10, 2018

Isn't this all an ESRI proprietary, closed source rendering stack? If so, there basically no way to run this on osmf hardware I'd say.

Ok, so then - Esri open sourcing the tools and style is what it would take? I mean, that's a good place to at least start a conversation.

@kocio-pl

This comment has been minimized.

Show comment
Hide comment
@kocio-pl

kocio-pl Jul 10, 2018

It wouldn't have to be default nor rendered on OSMF hardware, it could be just another layer just like Thunderforest ones (both using proprietary stack outside OSMF servers). Definitely worth talking with ESRI to know what's possible.

kocio-pl commented Jul 10, 2018

It wouldn't have to be default nor rendered on OSMF hardware, it could be just another layer just like Thunderforest ones (both using proprietary stack outside OSMF servers). Definitely worth talking with ESRI to know what's possible.

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Jul 10, 2018

Member

I don't think we're looking to just copy something, and a pure client side rendering is unlikely to be usable as the default style anyway. I imagine we would want to start with server side rendering of vector tiles and then experiment with moving the rendering to the client where possible.

But, to repeat myself for like the one billionth time, somebody needs to take on the job of evaluating the various available stacks and making a recommendation of what we should and what additional components (if any) need to be developed to glue them together.

Plus of course somebody needs to define both the extraction rules (ie what data to put in the vector tiles) and the rendering rules to be used with whatever stack is chosen.

Member

tomhughes commented Jul 10, 2018

I don't think we're looking to just copy something, and a pure client side rendering is unlikely to be usable as the default style anyway. I imagine we would want to start with server side rendering of vector tiles and then experiment with moving the rendering to the client where possible.

But, to repeat myself for like the one billionth time, somebody needs to take on the job of evaluating the various available stacks and making a recommendation of what we should and what additional components (if any) need to be developed to glue them together.

Plus of course somebody needs to define both the extraction rules (ie what data to put in the vector tiles) and the rendering rules to be used with whatever stack is chosen.

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Jul 10, 2018

Member

@kocio-pl well we are not setup at all for client side rendering at the moment so it would be a lot more complicated then the normal addition of a new tile layer.

Member

tomhughes commented Jul 10, 2018

@kocio-pl well we are not setup at all for client side rendering at the moment so it would be a lot more complicated then the normal addition of a new tile layer.

@kocio-pl

This comment has been minimized.

Show comment
Hide comment
@kocio-pl

kocio-pl Jul 10, 2018

I'm pretty sure that anything related to vector tiles will be non-standard, so I expect problems no matter which way we choose. I just think it's important to try, because it's the future of online maps.

kocio-pl commented Jul 10, 2018

I'm pretty sure that anything related to vector tiles will be non-standard, so I expect problems no matter which way we choose. I just think it's important to try, because it's the future of online maps.

@pnorman

This comment has been minimized.

Show comment
Hide comment
@pnorman

pnorman Jul 12, 2018

Collaborator

To return back to the topic - what would it take to make this the default style on openstreetmap.org? I think the Esri team has done a great job!

Their layer? The biggest is they'd need to have minutely updates, not every few weeks. The default layer needs fast updates because it's part of the feedback loop. For the default I'd also want a style with an established record of responding to user enhancement requests. I don't see it happening because they're aiming at a market where those aren't concerns. Going to minutely updates would probably double their hardware cost, and I can't even find a repo with their style in an editable form.

Collaborator

pnorman commented Jul 12, 2018

To return back to the topic - what would it take to make this the default style on openstreetmap.org? I think the Esri team has done a great job!

Their layer? The biggest is they'd need to have minutely updates, not every few weeks. The default layer needs fast updates because it's part of the feedback loop. For the default I'd also want a style with an established record of responding to user enhancement requests. I don't see it happening because they're aiming at a market where those aren't concerns. Going to minutely updates would probably double their hardware cost, and I can't even find a repo with their style in an editable form.

@kocio-pl

This comment has been minimized.

Show comment
Hide comment
@kocio-pl

kocio-pl Jul 14, 2018

I intially thought this might be some (direct) conversion to JSON of the existing openstreetmap-carto style, and ESRI using their own JSON style editor to potentially fine tune and modify the result.

I try to get in touch with them to talk more (@mboeringa maybe you have direct contact to them?), but found this link to the repo and it looks to me like quite different implementation of OSM Carto look and feel:

https://www.arcgis.com/sharing/rest/content/items/3e1a00aeae81496587988075fe529f71/resources/styles/root.json?f=pjson

kocio-pl commented Jul 14, 2018

I intially thought this might be some (direct) conversion to JSON of the existing openstreetmap-carto style, and ESRI using their own JSON style editor to potentially fine tune and modify the result.

I try to get in touch with them to talk more (@mboeringa maybe you have direct contact to them?), but found this link to the repo and it looks to me like quite different implementation of OSM Carto look and feel:

https://www.arcgis.com/sharing/rest/content/items/3e1a00aeae81496587988075fe529f71/resources/styles/root.json?f=pjson

@slibby

This comment has been minimized.

Show comment
Hide comment
@slibby

slibby Jul 29, 2018

I'm happy to help answer some questions from the Esri side of things and I'm glad to see the interest in the basemap work that our team has done. There are several parts to the "stack" that are worth referencing:

tl;dr: OSM minutely changesets -> PostGIS -> ArcGIS Pro -> Vector Tiles (nearly-mapbox-spec).

  • Where the changesets come from and how often they're applied
  • Where the data is stored in "current" form
  • How the vector tiles are designed and styled
  • How the vector tiles themselves are generated and then hosted

For the first half of that list, we're using a very common approach - changesets direct from OSM, minutely diffs (I believe) - these are then passed in as updates into a PostGIS database of the Planet, created and updated using a variety of commonly-used OSM-built tools. I don't know the details 100% but I believe one of my colleagues (@skmoore) is presenting on it at SOTM US in October.

Now that we have a full, minutely-updated dataset in PostGIS, the cartography was designed in ArcGIS Pro, and vector tiles were created using ArcGIS Pro. The cartographic design ends up in a style .json file that you can find linked just above this comment, and ArcGIS Pro generates a vector tile package of the tiles for the world. At this point is where the minutely updates are replaced by "less often" updates - with the goal, as our blog post said, of increasing that frequency. I will try to get my other colleagues to join in on this thread, or perhaps better -- publish a more detailed technical overview on a Github repository.

cc @jgravois @ajturner

slibby commented Jul 29, 2018

I'm happy to help answer some questions from the Esri side of things and I'm glad to see the interest in the basemap work that our team has done. There are several parts to the "stack" that are worth referencing:

tl;dr: OSM minutely changesets -> PostGIS -> ArcGIS Pro -> Vector Tiles (nearly-mapbox-spec).

  • Where the changesets come from and how often they're applied
  • Where the data is stored in "current" form
  • How the vector tiles are designed and styled
  • How the vector tiles themselves are generated and then hosted

For the first half of that list, we're using a very common approach - changesets direct from OSM, minutely diffs (I believe) - these are then passed in as updates into a PostGIS database of the Planet, created and updated using a variety of commonly-used OSM-built tools. I don't know the details 100% but I believe one of my colleagues (@skmoore) is presenting on it at SOTM US in October.

Now that we have a full, minutely-updated dataset in PostGIS, the cartography was designed in ArcGIS Pro, and vector tiles were created using ArcGIS Pro. The cartographic design ends up in a style .json file that you can find linked just above this comment, and ArcGIS Pro generates a vector tile package of the tiles for the world. At this point is where the minutely updates are replaced by "less often" updates - with the goal, as our blog post said, of increasing that frequency. I will try to get my other colleagues to join in on this thread, or perhaps better -- publish a more detailed technical overview on a Github repository.

cc @jgravois @ajturner

@systemed

This comment has been minimized.

Show comment
Hide comment
@systemed

systemed Jul 30, 2018

We had a really productive (and very well-attended) Birds of a Feather at State of the Map on Sunday 29th, on the subject of getting vector tiles on osm.org. Huge thanks to everyone who attended - it was great to see so much enthusiasm for it.

(Some prior reading before the session: http://blog.systemed.net/post/15, https://www.openstreetmap.org/user/Zverik/diary/44506, https://twitter.com/richardf/status/1023553727389683712 .)

A few takeaways from the session:

  • Universal agreement that we want vector tiles on osm.org, sooner rather than later!
  • We are not expecting to replace openstreetmap-carto, at least in the medium term. osm-carto has a strong community and its own particular strengths that are not easily replicated via vector tiles. Vector tiles and (raster) osm-carto would run in parallel, much as iD and P2 did/do. That means we can iterate the schema, styles and code rather than expecting v1 to equal osm-carto.
  • We are not intending to serve raster tiles from the vector stack, but solely deliver vector tiles to the client
  • Aim to serve as many OSM geometries/tags as possible within the confines of sensibly-sized tiles
  • Consider also serving an alternative 'lighter' schema for bandwidth-constrained scenarios, maybe as a later phase of development
  • Those with experience in the field (e.g. Paul and Kate) recommended considering tegola as the server
  • No strong views expressed on which renderer to use
  • People are generally happy with OSM's approach to tile usage (i.e. no heavy usage, no guarantee) and a vector tile stack may offer opportunities to enforce this through tokens etc.
  • Enthusiasm for a simple style editor/designer on osm.org
  • Hardware is doable - OSMF has several tileservers for osm-carto; it's possible that one of those could be repurposed (or a new one identfied) for serving vector tiles
  • Paul suggested that JS integration into osm.org should be less difficult than I'd anticipated

So, next steps?

To me one obvious starting point is a very rough, v0.00001 schema for osm.org vector tiles, which we can use to spin up sample deployments and start to develop against.

In due course we will probably need to move away from this operations ticket and set up a separate repo, but I hope the ops team will indulge us for a short while yet!

systemed commented Jul 30, 2018

We had a really productive (and very well-attended) Birds of a Feather at State of the Map on Sunday 29th, on the subject of getting vector tiles on osm.org. Huge thanks to everyone who attended - it was great to see so much enthusiasm for it.

(Some prior reading before the session: http://blog.systemed.net/post/15, https://www.openstreetmap.org/user/Zverik/diary/44506, https://twitter.com/richardf/status/1023553727389683712 .)

A few takeaways from the session:

  • Universal agreement that we want vector tiles on osm.org, sooner rather than later!
  • We are not expecting to replace openstreetmap-carto, at least in the medium term. osm-carto has a strong community and its own particular strengths that are not easily replicated via vector tiles. Vector tiles and (raster) osm-carto would run in parallel, much as iD and P2 did/do. That means we can iterate the schema, styles and code rather than expecting v1 to equal osm-carto.
  • We are not intending to serve raster tiles from the vector stack, but solely deliver vector tiles to the client
  • Aim to serve as many OSM geometries/tags as possible within the confines of sensibly-sized tiles
  • Consider also serving an alternative 'lighter' schema for bandwidth-constrained scenarios, maybe as a later phase of development
  • Those with experience in the field (e.g. Paul and Kate) recommended considering tegola as the server
  • No strong views expressed on which renderer to use
  • People are generally happy with OSM's approach to tile usage (i.e. no heavy usage, no guarantee) and a vector tile stack may offer opportunities to enforce this through tokens etc.
  • Enthusiasm for a simple style editor/designer on osm.org
  • Hardware is doable - OSMF has several tileservers for osm-carto; it's possible that one of those could be repurposed (or a new one identfied) for serving vector tiles
  • Paul suggested that JS integration into osm.org should be less difficult than I'd anticipated

So, next steps?

To me one obvious starting point is a very rough, v0.00001 schema for osm.org vector tiles, which we can use to spin up sample deployments and start to develop against.

In due course we will probably need to move away from this operations ticket and set up a separate repo, but I hope the ops team will indulge us for a short while yet!

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Jul 30, 2018

Member

I'm not sure that I agree that we decided that "We are not intending to serve raster tiles from the vector stack, but solely deliver vector tiles to the client" actually.

Rather my memory is that was a strong feeling that while we may not use the capability initially we need to preserve it as an option so that we can support older clients one a vector style reaches the point where we can consider making it the default view.

Member

tomhughes commented Jul 30, 2018

I'm not sure that I agree that we decided that "We are not intending to serve raster tiles from the vector stack, but solely deliver vector tiles to the client" actually.

Rather my memory is that was a strong feeling that while we may not use the capability initially we need to preserve it as an option so that we can support older clients one a vector style reaches the point where we can consider making it the default view.

@systemed

This comment has been minimized.

Show comment
Hide comment
@systemed

systemed Jul 30, 2018

Thanks, yep you're right.

systemed commented Jul 30, 2018

Thanks, yep you're right.

@kocio-pl

This comment has been minimized.

Show comment
Hide comment
@kocio-pl

kocio-pl Jul 30, 2018

I have set a thread on a forum to discuss vectors:

https://forum.openstreetmap.org/viewtopic.php?id=62160

which is technically more inclusive than GitHub (it's an OSMF asset with standard mapper login) and neutral (OSM Carto has it's own vector-related tickets - https://github.com/gravitystorm/openstreetmap-carto/projects/5).

Could we move there with general vector issues and leave this ticket for implementation details for example?

kocio-pl commented Jul 30, 2018

I have set a thread on a forum to discuss vectors:

https://forum.openstreetmap.org/viewtopic.php?id=62160

which is technically more inclusive than GitHub (it's an OSMF asset with standard mapper login) and neutral (OSM Carto has it's own vector-related tickets - https://github.com/gravitystorm/openstreetmap-carto/projects/5).

Could we move there with general vector issues and leave this ticket for implementation details for example?

@theel0ja

This comment has been minimized.

Show comment
Hide comment
@theel0ja

theel0ja Aug 2, 2018

This also would allow better indoor maps with floor selector instead of just piling levels on top of each other.

theel0ja commented Aug 2, 2018

This also would allow better indoor maps with floor selector instead of just piling levels on top of each other.

@kocio-pl

This comment has been minimized.

Show comment
Hide comment
@kocio-pl

kocio-pl Aug 13, 2018

@slibby I wait to hear more informations about it. Any plans about publishing repo?

kocio-pl commented Aug 13, 2018

@slibby I wait to hear more informations about it. Any plans about publishing repo?

@matthijsmelissen

This comment has been minimized.

Show comment
Hide comment
@matthijsmelissen

matthijsmelissen Aug 13, 2018

well we are not setup at all for client side rendering at the moment so it would be a lot more complicated then the normal addition of a new tile layer.

@tomhughes What would it take, technically, to display the ESRI vector tiles on openstreetmap.org? Assume for now that we would be allowed to use their tiles. As far as I understand, this would only require some changes to the openstreetmap-website front-end code, correct? Would this be very complex?

matthijsmelissen commented Aug 13, 2018

well we are not setup at all for client side rendering at the moment so it would be a lot more complicated then the normal addition of a new tile layer.

@tomhughes What would it take, technically, to display the ESRI vector tiles on openstreetmap.org? Assume for now that we would be allowed to use their tiles. As far as I understand, this would only require some changes to the openstreetmap-website front-end code, correct? Would this be very complex?

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Aug 13, 2018

Member

How would I know?

My understanding from conversations at SOTM is that there's no sane way to do it - basically everybody said that leaflet doesn't really support vector tiles in a useful way so you need to use OpenLayers or Mapbox GL but that means having different layers using different frameworks which I'm not even going to consider as a reasonable solution.

The only alternative would be to switch everything back to OpenLayers, which will be a lot of work.

So basically, as far as I know (and I have little practical experience with vector tiles) there is no quick or easy way to do it.

Member

tomhughes commented Aug 13, 2018

How would I know?

My understanding from conversations at SOTM is that there's no sane way to do it - basically everybody said that leaflet doesn't really support vector tiles in a useful way so you need to use OpenLayers or Mapbox GL but that means having different layers using different frameworks which I'm not even going to consider as a reasonable solution.

The only alternative would be to switch everything back to OpenLayers, which will be a lot of work.

So basically, as far as I know (and I have little practical experience with vector tiles) there is no quick or easy way to do it.

@matthijsmelissen

This comment has been minimized.

Show comment
Hide comment
@matthijsmelissen

matthijsmelissen Aug 13, 2018

Thanks, I understand the problem now. Does anybody know if there is any work being done on the Leaflet side to support vector tiles? Are the Leaflet people even aware that we're running into this?

matthijsmelissen commented Aug 13, 2018

Thanks, I understand the problem now. Does anybody know if there is any work being done on the Leaflet side to support vector tiles? Are the Leaflet people even aware that we're running into this?

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Aug 13, 2018

Member

I think somebody said there's a plugin for leaflet but that it's not very good.

I imagine mapbox are concentrating on Mapbox GL now rather than Leaflet.

Member

tomhughes commented Aug 13, 2018

I think somebody said there's a plugin for leaflet but that it's not very good.

I imagine mapbox are concentrating on Mapbox GL now rather than Leaflet.

@theel0ja

This comment has been minimized.

Show comment
Hide comment

theel0ja commented Aug 13, 2018

@tomhughes

This comment has been minimized.

Show comment
Hide comment
@tomhughes

tomhughes Aug 13, 2018

Member

Like I say I'm not expert and I'm probably misrememebering everything.

Asking @pnorman would probably get you more sensible answers.

Member

tomhughes commented Aug 13, 2018

Like I say I'm not expert and I'm probably misrememebering everything.

Asking @pnorman would probably get you more sensible answers.

@theel0ja

This comment has been minimized.

Show comment
Hide comment
@theel0ja

theel0ja Aug 13, 2018

There is also Mapbox GL wrapper for Leaflet, but it doesn't have rotation, and performance isn't as good etc.

https://github.com/mapbox/mapbox-gl-leaflet

theel0ja commented Aug 13, 2018

There is also Mapbox GL wrapper for Leaflet, but it doesn't have rotation, and performance isn't as good etc.

https://github.com/mapbox/mapbox-gl-leaflet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment