Vector tiles #166

sdlime · 2017-04-06T04:04:07Z

Hi Thomas: I have something working with WMS source and a disk cache. Just wondering if you could take a peek. I'm thinking WMS is probably the most meaningful source, but other cache types would need support. Config would look something like:

<format name="MVT" type="RAW">
   <extension>mvt</extension>
   <mime_type>application/x-protobuf</mime_type>
</format>

<!-- WMS returns Mapbox Vector Tiles (.mvt) -->
<source name="test_wms_mvt" type="wms">
  <http>
    <url>http://mapserver.localhost/cgi-bin/mapserv?</url>
  </http>
  <getmap>
    <params>
      <LAYERS>sna,wma,lakes,major_roads</LAYERS>
      <FORMAT>application/x-protobuf</FORMAT>
      <MAP>test.map</MAP>
    </params>
  </getmap>
</source>

<tileset name="test_mvt">
  <source>test_wms_mvt</source>
  <grid>g</grid>
  <cache>disk</cache>
  <format>MVT</format>
</tileset>

--Steve

sdlime · 2017-04-06T14:58:14Z

I noticed it's not setting the right mime-type on requests. Still works but that's something I need to fix... --Steve

tbonfort

Thanks for starting this @sdlime . I'm surprised there's so little change needed, I think we need a few more checks for robustness, notably:

disallowing metatile support when using mvt
disallowing tile merging when one of the layers is mvt
going through the code and checking everywhere tile->raw_image is used to make sure the blob case is handled

… mime-type can be set.

sdlime · 2017-04-11T00:48:39Z

Thanks for the review. I was surprised too. Once I got the tile written to the disk cache the response piece just sort of worked (just fixed mime-type). I knew robustness would be an issue. Aside from that the approach looks reasonable? Could also (maybe) enable UTFGrids through this. I thought that had been done via GSoC but I guess not. --Steve

…leteness.

sdlime · 2017-04-26T23:56:34Z

@tbonfort, I tried and this approach does work for UTFGrid too, for example:

<format name="UTFGRID" type="RAW">
   <extension>json</extension>
   <mime_type>application/json</mime_type>
</format>

<!-- WMS returns UTFGrid data (.json) -->
<source name="test_wms_mvt" type="wms">
  <http>
    <url>http://mapserver.localhost/cgi-bin/mapserv?</url>
  </http>
  <getmap>
    <params>
      <LAYERS>sna,wma,lakes,major_roads</LAYERS>
      <FORMAT>application/json</FORMAT>
      <MAP>test.map</MAP>
    </params>
  </getmap>
</source>

<tileset name="test_utfgrid">
  <source>test_wms_utfgrid</source>
  <grid>g</grid>
  <cache>disk</cache>
  <format>UTFGRID</format>
</tileset>

Perhaps BLOB as a type is too narrow, maybe RAW instead?

Steve

…ta (like vector tiles). It can also be used for something like UTFGrid, so...

sdlime · 2017-04-27T01:20:26Z

Testing comment (will update as I can test cache types):

SQLite3: works without modification to cache_sqlite.c
BDB:

sdlime · 2017-05-09T01:01:22Z

@tbonfort I think it's easiest just to disable WMS support for tilesets referencing raw formats. I've updated service_wms.c to do that but wanted to get your thoughts. In getTile mode with one layer it can work but that's the only case where it seems to make sense. --Steve

sdlime · 2018-11-08T21:22:04Z

@tbonfort, just a bump - I know you're busy.

tbonfort · 2018-11-14T22:17:26Z

sorry about my inactivity... ok for disabling WMS for such layers. do you want a more in depth review? cheers

sdlime · 2018-11-15T17:30:07Z

No worries… I wouldn’t feel comfortable merging without a thorough review. I think I hit all the integration spots but I don’t know that code like you do. Would be nice to get a new MapCache release out with this patch and whatever else is sitting in master. I suspect I owe you some tests as well – some thoughts on what those should entail would be helpful. From: Thomas Bonfort [mailto:notifications@github.com] Sent: Wednesday, November 14, 2018 4:17 PM To: mapserver/mapcache <mapcache@noreply.github.com> Cc: Lime, Steve D (MNIT) <steve.lime@state.mn.us>; Mention <mention@noreply.github.com> Subject: Re: [mapserver/mapcache] Vector tiles (#166) sorry about my inactivity... ok for disabling WMS for such layers. do you want a more in depth review? cheers — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#166 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABhm-zs2OnbZmeOyxTNc2ZsLwPr8oAYeks5uvJZ3gaJpZM4M1F-2>.

tbonfort · 2019-01-18T15:16:59Z

lib/source_wms.c

@@ -92,7 +93,7 @@ void _mapcache_source_wms_render_map(mapcache_context *ctx, mapcache_map *map)
  mapcache_http_do_request(ctx,http,map->encoded_data,NULL,NULL);
  GC_CHECK_ERROR(ctx);

-  if(!mapcache_imageio_is_valid_format(ctx,map->encoded_data)) {
+  if(map->tileset->format->type != GC_RAW && !mapcache_imageio_is_valid_format(ctx,map->encoded_data)) {


not checking tileset->format!=NULL might be the cause of the crash in MapServer/MapServer#5596 (comment)_

tbonfort · 2019-01-18T15:17:14Z

lib/service_wms.c

@@ -728,7 +733,7 @@ void _mapcache_service_wms_parse_request(mapcache_context *ctx, mapcache_service
      goto proxies;
    } else {
      mapcache_tileset *tileset = mapcache_configuration_get_tileset(config,str);
-      if(!tileset) {
+      if(!tileset || tileset->format->type == GC_RAW) {


not checking tileset->format!=NULL might be the cause of the crash in MapServer/MapServer#5596 (comment)_

tbonfort · 2019-01-18T15:18:48Z

lib/core.c

@@ -502,6 +512,7 @@ mapcache_http_response *mapcache_core_get_map(mapcache_context *ctx, mapcache_re
  }

  /* compute the content-type */
+  ctx->log(ctx,MAPCACHE_DEBUG,"SDL: setting content type (2)");


stray debug message to remove

tbonfort · 2019-01-18T15:19:47Z

LGTM once the checks for tileset->format!=NULL are added (there are a couple few others I haven't ref'd directly in service_wms.c)

…w format (in response to tbonfort comments).

sdlime · 2019-02-13T16:57:47Z

LGTM once the checks for tileset->format!=NULL are added (there are a couple few others I haven't ref'd directly in service_wms.c)

I think I caught all the format != NULL checks... added a little helper function.

sdlime added 2 commits April 5, 2017 21:28

Basic source=wms, cache=disk support for blob format...

761022e

Removed my debugging calls.

23891f2

sdlime requested review from tbonfort and jmckenna April 6, 2017 04:04

tbonfort reviewed Apr 9, 2017

View reviewed changes

Make sure a format is set for non-image (i.e. blob) tiles so a proper…

cfe4bcb

… mime-type can be set.

sdlime added 2 commits April 12, 2017 20:23

Throw an error if a tileset defines blob format and metatiling.

be1c2eb

Expanded tileset blob/metatile check to include metabuffer - for comp…

8ba4cd8

…leteness.

Switched 'blob' to 'raw' since it can be used for more than binary da…

c7c5254

…ta (like vector tiles). It can also be used for something like UTFGrid, so...

sdlime added 2 commits May 8, 2017 19:23

Exclude raw layers from WMS GetCapabilities output.

3d6cd8a

Disable raw layers for getMap and getFeatureInfo WMS requests.

68a9140

mapcentia added a commit to mapcentia/dockerfiles that referenced this pull request Dec 13, 2018

Apply MapServer/mapcache#166

807e78e

$@fracaron$ fracaron mentioned this pull request Jan 4, 2019

MVT tile edge alignment issue (still) MapServer/MapServer#5596

Closed

tbonfort reviewed Jan 18, 2019

View reviewed changes

MapServer deleted a comment from fracaron Jan 25, 2019

sdlime added 2 commits February 13, 2019 10:52

Added a helper function to properly test if a tileset is using the ra…

3e0b391

…w format (in response to tbonfort comments).

Removed stray debug message.

d59dce0

sdlime merged commit c2fae80 into MapServer:master Feb 15, 2019

sdlime deleted the vector-tiles branch February 15, 2019 15:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vector tiles #166

Vector tiles #166

sdlime commented Apr 6, 2017 •

edited

sdlime commented Apr 6, 2017 •

edited

tbonfort left a comment

sdlime commented Apr 11, 2017

sdlime commented Apr 26, 2017 •

edited

sdlime commented Apr 27, 2017

sdlime commented May 9, 2017

sdlime commented Nov 8, 2018

tbonfort commented Nov 14, 2018

sdlime commented Nov 15, 2018 via email

tbonfort Jan 18, 2019

tbonfort Jan 18, 2019

tbonfort Jan 18, 2019

tbonfort commented Jan 18, 2019 •

edited

sdlime commented Feb 13, 2019

Vector tiles #166

Vector tiles #166

Conversation

sdlime commented Apr 6, 2017 • edited

sdlime commented Apr 6, 2017 • edited

tbonfort left a comment

Choose a reason for hiding this comment

sdlime commented Apr 11, 2017

sdlime commented Apr 26, 2017 • edited

sdlime commented Apr 27, 2017

sdlime commented May 9, 2017

sdlime commented Nov 8, 2018

tbonfort commented Nov 14, 2018

sdlime commented Nov 15, 2018 via email

tbonfort Jan 18, 2019

Choose a reason for hiding this comment

tbonfort Jan 18, 2019

Choose a reason for hiding this comment

tbonfort Jan 18, 2019

Choose a reason for hiding this comment

tbonfort commented Jan 18, 2019 • edited

sdlime commented Feb 13, 2019

sdlime commented Apr 6, 2017 •

edited

sdlime commented Apr 6, 2017 •

edited

sdlime commented Apr 26, 2017 •

edited

tbonfort commented Jan 18, 2019 •

edited