Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vector tiles #166

Merged
merged 10 commits into from Feb 15, 2019

Conversation

@sdlime
Copy link
Member

commented Apr 6, 2017

Hi Thomas: I have something working with WMS source and a disk cache. Just wondering if you could take a peek. I'm thinking WMS is probably the most meaningful source, but other cache types would need support. Config would look something like:

<format name="MVT" type="RAW">
   <extension>mvt</extension>
   <mime_type>application/x-protobuf</mime_type>
</format>

<!-- WMS returns Mapbox Vector Tiles (.mvt) -->
<source name="test_wms_mvt" type="wms">
  <http>
    <url>http://mapserver.localhost/cgi-bin/mapserv?</url>
  </http>
  <getmap>
    <params>
      <LAYERS>sna,wma,lakes,major_roads</LAYERS>
      <FORMAT>application/x-protobuf</FORMAT>
      <MAP>test.map</MAP>
    </params>
  </getmap>
</source>

<tileset name="test_mvt">
  <source>test_wms_mvt</source>
  <grid>g</grid>
  <cache>disk</cache>
  <format>MVT</format>
</tileset>

--Steve

@sdlime sdlime requested review from tbonfort and jmckenna Apr 6, 2017

@sdlime

This comment has been minimized.

Copy link
Member Author

commented Apr 6, 2017

I noticed it's not setting the right mime-type on requests. Still works but that's something I need to fix... --Steve

@tbonfort
Copy link
Member

left a comment

Thanks for starting this @sdlime . I'm surprised there's so little change needed, I think we need a few more checks for robustness, notably:

  • disallowing metatile support when using mvt
  • disallowing tile merging when one of the layers is mvt
  • going through the code and checking everywhere tile->raw_image is used to make sure the blob case is handled
@sdlime

This comment has been minimized.

Copy link
Member Author

commented Apr 11, 2017

Thanks for the review. I was surprised too. Once I got the tile written to the disk cache the response piece just sort of worked (just fixed mime-type). I knew robustness would be an issue. Aside from that the approach looks reasonable? Could also (maybe) enable UTFGrids through this. I thought that had been done via GSoC but I guess not. --Steve

@sdlime

This comment has been minimized.

Copy link
Member Author

commented Apr 26, 2017

@tbonfort, I tried and this approach does work for UTFGrid too, for example:

<format name="UTFGRID" type="RAW">
   <extension>json</extension>
   <mime_type>application/json</mime_type>
</format>

<!-- WMS returns UTFGrid data (.json) -->
<source name="test_wms_mvt" type="wms">
  <http>
    <url>http://mapserver.localhost/cgi-bin/mapserv?</url>
  </http>
  <getmap>
    <params>
      <LAYERS>sna,wma,lakes,major_roads</LAYERS>
      <FORMAT>application/json</FORMAT>
      <MAP>test.map</MAP>
    </params>
  </getmap>
</source>

<tileset name="test_utfgrid">
  <source>test_wms_utfgrid</source>
  <grid>g</grid>
  <cache>disk</cache>
  <format>UTFGRID</format>
</tileset>

Perhaps BLOB as a type is too narrow, maybe RAW instead?

Steve

Switched 'blob' to 'raw' since it can be used for more than binary da…
…ta (like vector tiles). It can also be used for something like UTFGrid, so...
@sdlime

This comment has been minimized.

Copy link
Member Author

commented Apr 27, 2017

Testing comment (will update as I can test cache types):

  1. SQLite3: works without modification to cache_sqlite.c
  2. BDB:
@sdlime

This comment has been minimized.

Copy link
Member Author

commented May 9, 2017

@tbonfort I think it's easiest just to disable WMS support for tilesets referencing raw formats. I've updated service_wms.c to do that but wanted to get your thoughts. In getTile mode with one layer it can work but that's the only case where it seems to make sense. --Steve

@sdlime

This comment has been minimized.

Copy link
Member Author

commented Nov 8, 2018

@tbonfort, just a bump - I know you're busy.

@tbonfort

This comment has been minimized.

Copy link
Member

commented Nov 14, 2018

sorry about my inactivity... ok for disabling WMS for such layers. do you want a more in depth review? cheers

@sdlime

This comment has been minimized.

Copy link
Member Author

commented Nov 15, 2018

mapcentia added a commit to mapcentia/dockerfiles that referenced this pull request Dec 13, 2018

@@ -92,7 +93,7 @@ void _mapcache_source_wms_render_map(mapcache_context *ctx, mapcache_map *map)
mapcache_http_do_request(ctx,http,map->encoded_data,NULL,NULL);
GC_CHECK_ERROR(ctx);

if(!mapcache_imageio_is_valid_format(ctx,map->encoded_data)) {
if(map->tileset->format->type != GC_RAW && !mapcache_imageio_is_valid_format(ctx,map->encoded_data)) {

This comment has been minimized.

Copy link
@tbonfort

tbonfort Jan 18, 2019

Member

not checking tileset->format!=NULL might be the cause of the crash in mapserver/mapserver#5596 (comment)_

@@ -728,7 +733,7 @@ void _mapcache_service_wms_parse_request(mapcache_context *ctx, mapcache_service
goto proxies;
} else {
mapcache_tileset *tileset = mapcache_configuration_get_tileset(config,str);
if(!tileset) {
if(!tileset || tileset->format->type == GC_RAW) {

This comment has been minimized.

Copy link
@tbonfort

tbonfort Jan 18, 2019

Member

not checking tileset->format!=NULL might be the cause of the crash in mapserver/mapserver#5596 (comment)_

lib/core.c Outdated
@@ -502,6 +512,7 @@ mapcache_http_response *mapcache_core_get_map(mapcache_context *ctx, mapcache_re
}

/* compute the content-type */
ctx->log(ctx,MAPCACHE_DEBUG,"SDL: setting content type (2)");

This comment has been minimized.

Copy link
@tbonfort

tbonfort Jan 18, 2019

Member

stray debug message to remove

@tbonfort

This comment has been minimized.

Copy link
Member

commented Jan 18, 2019

LGTM once the checks for tileset->format!=NULL are added (there are a couple few others I haven't ref'd directly in service_wms.c)

@mapserver mapserver deleted a comment from fracaron Jan 25, 2019

@mapserver mapserver deleted a comment from fracaron Jan 25, 2019

sdlime added some commits Feb 13, 2019

Added a helper function to properly test if a tileset is using the ra…
…w format (in response to tbonfort comments).
@sdlime

This comment has been minimized.

Copy link
Member Author

commented Feb 13, 2019

LGTM once the checks for tileset->format!=NULL are added (there are a couple few others I haven't ref'd directly in service_wms.c)

I think I caught all the format != NULL checks... added a little helper function.

@sdlime sdlime merged commit c2fae80 into mapserver:master Feb 15, 2019

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@sdlime sdlime deleted the sdlime:vector-tiles branch Feb 15, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.