MapServer as WMS client, spaces in layer names on upstream server not supported. #5232

bblammo · 2016-02-04T19:10:18Z

For more background see these threads:

http://osgeo-org.1560.x6.nabble.com/Allowing-spaces-in-layer-names-WMS-WFS-td5213960.html
https://lists.osgeo.org/pipermail/mapserver-users/2014-August/076870.html
https://lists.osgeo.org/pipermail/mapserver-users/2014-August/076886.html

Here is some minimal info on using the Census (Arc) server WMS, The server is using spaces in both the NAME and TITLE tags, which MapServer doesn't seem to want to push through.

So, as an example, using this WMS:

http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?

Specifically, two layers, “Census Tracts” and “Census Tracts Labels”

I can open each of these service calls in just about everything, except Mapserver as a WMS client. I’m assuming the problem is in how MapServer is sending it’s request.

I've tried encoding in the MapServer request string, as a "+", and " " and as "%02". These last two seem to work when directly hitting the WMS, but MapServer is not translating any of them through it's request.

Here is a test mapfile I was using:

Mapfile

   MAP
        LAYER
             NAME "Census Tracts"
             TYPE RASTER
             STATUS DEFAULT
             CONNECTION
   "http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?"

             CONNECTIONTYPE WMS
             METADATA
                  "wms_srs" "EPSG:4326"
                  "wms_name" "Census Tracts"
                  "wms_server_version" "1.1.1"
                  #"wms_server_version" "1.3.0"
                  "wms_format" "image/png24"

                  "ows_title" "Census Tracts"
                  #"gml_include_items" "all"
             END

             PROJECTION
                  "init=epsg:4326"
             END

             TEMPLATE "dummy"
        END #layer

   END

The text was updated successfully, but these errors were encountered:

jrahkonen · 2016-02-04T21:02:02Z

ArcGIS server requires that space is url-encoded as "%20" but it does not accept space encoded as "+". There has been a ticket in the ESRI ticket tracker about accepting both "%20" and "+" because also the http standard accepts them both but the resolution was "won't fix".

The issue code and description at ESRI was

[#NIM104588 For layer names having spaces, the GetMap request URL for ArcGIS WMS services requires %20 to be used in place of spaces instead of a '+' sign.]

jmckenna · 2016-02-04T23:41:33Z

Using your test layer, you can see the actual WMS GetMap request generated by MapServer through the commandline, by using the shp2img utility with the parameter "-all_debug 3", which shows that MapServer (testing with master) replaces the space with a "+" in the layer name:

http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?  
LAYERS=Census+Tracts
&REQUEST=GetMap
&SERVICE=WMS
&FORMAT=image/png24
&STYLES=
&HEIGHT=300
&VERSION=1.1.1
&SRS=EPSG:4326
&WIDTH=400
&BBOX=-180.451127819549,-135.338345864662,180.451127819549,135.338345864662
&TRANSPARENT=TRUE
&EXCEPTIONS=application/vnd.ogc.se_inimage

tchaddad · 2016-02-05T00:14:16Z

In one of the threads above the user reports success just by removing the detection of the space and recompiling:

https://lists.osgeo.org/pipermail/mapserver-users/2014-August/076879.html

Seems like there could be some unintentional effects to taking that route, but...

bblammo · 2016-02-05T22:49:47Z

A valid Census Server request:

http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?LAYERS=Census%20Tracts,Census%20Tracts%20Labels&REQUEST=GetMap&SERVICE=WMS&FORMAT=image/png24&STYLES=&HEIGHT=600&VERSION=1.1.1&SRS=EPSG:4326&WIDTH=800&BBOX=-93.239,44.892,-92.977,44.993&TRANSPARENT=TRUE&EXCEPTIONS=application/vnd.ogc.se_inimage

jmckenna · 2016-02-11T15:46:05Z

Bob, Tanya, Jukka, all,

I've applied the change mentioned in that thread, and created a very "experimental" build of MS4W for our testing of this. http://www.ms4w.com/release/experimental/ms4w_3.1.3dev_experimental.zip

I've now successfully used the following mapfile, with a space in the client layer name, to generate a map image:

MAP
NAME "test"
STATUS ON
SIZE 400 300
EXTENT -93.239 44.892 -92.977 44.993
UNITS DD
IMAGECOLOR 255 255 255

LAYER
  NAME "census_tracts"
  TYPE RASTER
  STATUS ON
  CONNECTION "http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?"
  CONNECTIONTYPE WMS
  METADATA
      "wms_srs"         "EPSG:4326"
      "wms_name"        "Census Tracts"
      "wms_server_version"  "1.1.1"
      "wms_format"      "image/png24"
  END
  PROJECTION
  "init=epsg:4326"
  END
END #layer 

END #map

This generated a valid map image of the tracts. And with "all_debug 3" switch for shp2img I can see the correct GetMap request generated by MapServer:

http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?LAYERS=Census%20Tracts&REQUEST=GetMap&SERVICE=WMS&FORMAT=image/png24&STYLES=&HEIGHT=300&VERSION=1.1.1&SRS=EPSG:4326&WIDTH=400&BBOX=-93.239328320802,44.8440037593985,-92.976671679198,45.0409962406015&TRANSPARENT=TRUE&EXCEPTIONS=application/vnd.ogc.se_inimage

Give it a try, such as with GeoMOOSE, and any other ways you can think this change may affect things.

-jeff

jmckenna · 2016-02-19T18:38:36Z

Any testing feedback? I notice that there were 3 downloads of the experimental build.

MapServer/MapServer#5232

tbonfort · 2016-02-24T14:18:11Z

Applied to branch-7-0 in b6fb921 , autotests updated in MapServer/msautotest_DEPRECATED@c54b0b9

tbonfort · 2016-02-24T14:23:20Z

While this is not an issue with mapserver per-se, and thus not a fix that should rightfully belong in a maintenance release, after discussion we decided that the risks of this fix were rather low, and that it solved a real inter-operability issue. Thus the inclusion in 7.0.1 instead of 7.2

jmckenna · 2016-02-24T15:26:32Z

interesting. Not sure why my pull request wasn't accepted. ok sorry for the noise, glad you all decided.

tbonfort · 2016-02-24T15:33:53Z

Maybe because it was full of whitespace changes in areas unrelated to this fix ?

jmckenna · 2016-02-24T15:40:19Z

ouch, ok you didn't let me know. k sorry for the noise.

bblammo · 2016-02-25T21:53:06Z

So, I got my Census layer to work, mostly. Thanks everyone!!

One little follow on item though related to legends, that I figured I should ask about here is.

I'm getting the "Parameter 'layers' contains unacceptable layer names." error when trying to call for a pass through legend graphic. My question is, is there another location in the code that is calling for a Legend graphic separately that also needs to be tweaked for the white space(s)?

Thanks

Just to complicate matters, I'mn actually calling for two different layers in my client WMS mapfile:

"Census Tracts,Census Tracts Labels"
NOTE: Mapserver doesn't like a space after the comma above, I believe it's intereted as part of the second layer name. Should this be added in the DOCs somewhere?

jmckenna · 2016-02-25T22:35:57Z

Can you give some steps to reproduce? (in other words, how are you generating the legend graphic call) I believe it is the application/client-side that generates the GetLegendGraphic url, not MapServer (but I've been wrong before ha). If you explain how to reproduce, that makes it easier to understand.

bblammo · 2016-02-25T22:38:28Z

GetLegendGraphic call to (our) Mapserver that is a client to the Census
WMSservice. Pass through legend request.

Bobb
On Feb 25, 2016 4:36 PM, "Jeff McKenna" notifications@github.com wrote:

Can you give some steps to reproduce? (in other words, how are you
generating the legend graphic call) I believe it is the
application/client-side that generates the GetLegendGraphic url, not
MapServer (but I've been wrong before ha). If you explain how to reproduce,
that makes it easier to understand.

—
Reply to this email directly or view it on GitHub
#5232 (comment)
.

jmckenna · 2016-02-25T22:50:43Z

Trying to reproduce here, that is why I ask. How are you generating the GetLegendGraphic call? (can you tell me the steps so I can reproduce locally?) Example, use the following mapfile, then do this command, etc.

jmckenna · 2016-02-25T22:53:26Z

Sorry, confusing.

jmckenna · 2016-02-25T23:14:56Z

Here is a working GetLegendGraphic request for the layer "Census Tracts":

https://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WmsServer?request=GetLegendGraphic%26version=1.1.1%26format=image/png%26layer=Census%20Tracts

tchaddad · 2016-02-25T23:18:51Z

Sounds like the failure is on @bblammo's server which is sitting in between his application and the census service. So the failure is an incoming GetLegendGraphic request to his Mapserver, not getting passed on to the Census server...

jmckenna · 2016-02-25T23:22:25Z

In simple CGI mode, locally I am using the same test mapfile, with "Census Tracts" layer, and I get a blank white legend graphic from MapServer, no error. I wonder how to reproduce @bblammo's error of ""Parameter 'layers' contains unacceptable layer names."

jmckenna · 2016-02-25T23:23:57Z

Actually here is the png legend image generated by my local MapServer:

jmckenna · 2016-02-25T23:33:20Z

Oh!!!

http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?LAYERS=Census%20Tracts,Census%20Tract%20Labels&REQUEST=GetMap&SERVICE=WMS&FORMAT=image/png24&STYLES=&HEIGHT=300&VERSION=1.1.1&SRS=EPSG:4326&WIDTH=400&BBOX=-93.239328320802,44.8440037593985,-92.976671679198,45.0409962406015&TRANSPARENT=TRUE&EXCEPTIONS=application/vnd.ogc.se_inimage

(save that file and name it "WMSServer.png" and open it, bingo, it is ArcGIS's "Parameter 'layers' contains unacceptable layer names" error)

I have never used 2 layers in a single WMS client layer before. If you separate them into 2 layers this error will not occur.

jmckenna · 2016-02-25T23:37:59Z

How would a MapServer generate a map image, for 2 layers at the same time, and combine that into one single Legend image? I think that is asking for magic :)

jmckenna · 2016-02-26T00:01:19Z

I'm not sure how MapServer would handle a GetLegendGraphic request for multiple layers into one image, but I'll let others comment now. Hope my debugging helped others.

jmckenna · 2016-02-26T00:19:40Z

Could it actually be that ArcGIS Server is the one with the problem, in handling multiple layer names with spaces? My local CGI viewer actually gives that error for the GetMap call. (the legend is blank, but it is the GetMap request that causes the error) Ok sorry for so many messages ha.

jmckenna · 2016-02-26T15:30:46Z

To reproduce this issue:

use this mapfile (single WMSclient layer that references 2 remote WMS layers each containing spaces):

MAP
NAME "test"
STATUS ON
SIZE 400 300
EXTENT -98.182 31.650 -98.032 31.750
UNITS DD
IMAGECOLOR 255 255 255

LAYER
  NAME "census_tracts"
  TYPE RASTER
  STATUS DEFAULT
  CONNECTION "http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?"
  CONNECTIONTYPE WMS
  METADATA
      "wms_srs"         "EPSG:4326"
      "wms_name"        "Census Tracts,Census Tract Labels"
      "wms_server_version"  "1.1.1"
      "wms_format"      "image/png24"
  END
END #layer 

END #map

use shp2img such as:

shp2img -m wms-client-space.map -o ttt.png -all_debug 5

a map image is returned by ArcGIS Server with the error:

the GetMap request generated by MapServer is:

http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?LAYERS=Census%20Tracts,Census%20Tract%20Labels&REQUEST=GetMap&SERVICE=WMS&FORMAT=image/png24&STYLES=&HEIGHT=300&VERSION=1.1.1&SRS=EPSG:4326&WIDTH=400&BBOX=-98.1821879699248,31.6436090225564,-98.0318120300752,31.7563909774436&TRANSPARENT=TRUE&EXCEPTIONS=application/vnd.ogc.se_inimage

(save that returned file as 'MapServer.png' and open it, to view the error image)

jmckenna · 2016-02-26T15:42:03Z

Ah solved, it was a mistake in one of the names - "Census Tracts Labels" (tracts not tract). Now everything works (CGI mode, shp2img, and a proper blank legend image is returned). Likely @bblammo might have made same mistake. I think this ticket is closed now.

working mapfile:

MAP
NAME "test"
STATUS ON
SIZE 400 300
EXTENT -98.182 31.650 -98.032 31.750
UNITS DD
IMAGECOLOR 255 255 255

LAYER
  NAME "census_tracts"
  TYPE RASTER
  STATUS DEFAULT
  CONNECTION "http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?"
  CONNECTIONTYPE WMS
  METADATA
      "wms_srs"         "EPSG:4326"
      "wms_name"        "Census Tracts,Census Tracts Labels"
      "wms_server_version"  "1.1.1"
      "wms_format"      "image/png24"
  END
END #layer 

END #map

working request generated by MapServer:

http://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WMSServer?LAYERS=Census%20Tracts,Census%20Tracts%20Labels&REQUEST=GetMap&SERVICE=WMS&FORMAT=image/png24&STYLES=&HEIGHT=300&VERSION=1.1.1&SRS=EPSG:4326&WIDTH=400&BBOX=-98.1821879699248,31.6436090225564,-98.0318120300752,31.7563909774436&TRANSPARENT=TRUE&EXCEPTIONS=application/vnd.ogc.se_inimage

correct image returned by ArcGIS Server:

tchaddad · 2016-02-26T16:52:03Z

@bblammo will have to confirm - his second issue was about GetLegendGraphic, but perhaps his expectation of requesting 2 images in one request was not a realistic one.

bblammo · 2016-02-26T16:54:29Z

The map image returns just fine. Just trying to figure out how to display
a legend for the remote WMS service. I think a can fake one in the
mapfile, bur it would really be best to pass through from the upstream
server if possible.

Bobb
On Feb 26, 2016 10:52 AM, "TC Haddad" notifications@github.com wrote:

@bblammo https://github.com/bblammo will have to confirm - his second
issue was about GetLegendGraphic, but perhaps his expectation of requesting
2 images in one request was not a realistic one.

—
Reply to this email directly or view it on GitHub
#5232 (comment)
.

jmckenna · 2016-02-26T17:05:53Z

As I mentioned, MapServer returns the legend fine (well a blank image in my testing). Regarding a GetLegendGraphic call with a proper legend graphic returned from the remote WMS server, this is when I always turn to client side code (OpenLayers, GeoMOOSE, GeoExt,...) to generate a nice GetLegendGraphic call for each layer (notice I say each layer, not for 2 combined layers). I was trying to ask earlier what you are using to generate that GetLegendGraphic url. It's pretty simple in javascript once you have the layer name. You can get a proper GetLegendGraphic url by executing a GetCapabilities request, and opening it in your text editor and finding the layer you need, the GetLegendGraphic url is listed there, such as:

https://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WmsServer?request=GetLegendGraphic%26version=1.1.1%26format=image/png%26layer=Census%20Tracts

tchaddad · 2016-02-26T17:08:36Z

If I understand correctly, he is making a GetLegendGraphic request to his Mapserver, which he is expecting Mapserver to pass on to the Census server. Somewhere in that passing, his request is failing, possibly due to Mapserver not correctly handling the passing of the name with spaces.

But that is just how I'm reading the thread...

bblammo · 2016-02-26T17:08:50Z

Iowa just hoping there was a way to automate the legend call. We set up
all our map files for WMS etc. So being able to do the pass through negates
the need to manually set for each off site layer. We have a lot if these
with a lot more potentially being added.

Bobb
On Feb 26, 2016 11:06 AM, "Jeff McKenna" notifications@github.com wrote:

As I mentioned, MapServer returns the legend fine (well a blank image in
my testing). Regarding a GetLegendGraphic call with a proper legend graphic
returned from the remote WMS server, this is when I always turn to client
side code (OpenLayers, GeoMOOSE, GeoExt,...) to generate a nice
GetLegendGraphic call for each layer (notice I say each layer, not for 2
combined layers). I was trying to ask earlier what you are using to
generate that GetLegendGraphic url. It's pretty simple in javascript once
you have the layer name. You can get a proper GetLegendGraphic url by
executing a GetCapabilities request, and opening it in your text editor and
finding the layer you need, the GetLegendGraphic url is listed there, such
as:

https://tigerweb.geo.census.gov/arcgis/services/TIGERweb/tigerWMS_ACS2015/MapServer/WmsServer?request=GetLegendGraphic%26version=1.1.1%26format=image/png%26layer=Census%20Tracts

—
Reply to this email directly or view it on GitHub
#5232 (comment)
.

bblammo · 2016-02-26T17:10:00Z

That's it in nutshell. :c)

Bobb
On Feb 26, 2016 11:08 AM, "TC Haddad" notifications@github.com wrote:

If I understand correctly, he is making a GetLegendGraphic request to his
Mapserver, which he is expecting Mapserver to pass on to the Census server.
Somewhere in that passing, his request is failing, possibly due to
Mapserver not correctly handling the passing of the name with spaces.

But that is just how I'm reading the thread...

—
Reply to this email directly or view it on GitHub
#5232 (comment)
.

tchaddad · 2016-02-26T17:11:31Z

So it is possible when we 'fixed' the GetMap portion of Mapserver to work with spaces, we missed another location where GetLegendGraphic also needs a similar fix applied...

jmckenna · 2016-02-26T17:15:52Z

Not by all my testing (above). I was trying to show that no errors are generated, in MapServer CGI mode with a legend (my images above with legends).

I've never seen MapServer CGI generate a GetLegendGraphic properly from a remote WMS service by itself. Do you have other examples we can test (other WMS servers, no spaces in layers, simple short mapfile)?

You might also want to bring this issue, with a short test mapfile (one small mapfile that does generate your GetLegendGraphic, and one that does not) maybe to the larger MapServer-users list, where it isn't just me answering ha. Not sure.

jmckenna · 2016-02-26T17:19:15Z

@tchaddad definitely possible. I think it would help to have one tiny mapfile with one layer from a WMS service and steps to get a proper GetLegendGraphic. And maybe more eyes on this (mailing list). But I'm not sure.

tchaddad · 2016-02-26T17:19:25Z

Yeah, I agree Jeff, a direct legend request is definitely working with a space in the name

I'm not 100% certain, but I think "pass through" requests are possible using MapCache, so I can see why Bobb was trying to get this to work.

I think I have a service where I can test this, will report back...

jmckenna · 2016-02-26T22:35:24Z

I've updated the WMS Client document and included this same TIGER example with 2 layers, and also a note about the comma must not contain spaces around it. http://www.mapserver.org/ogc/wms_client.html#adding-a-wms-layer

tchaddad · 2016-02-26T23:22:53Z

OK - after some testing, I think I can report 2 things:

"pass through" GetLegendGraphic calls are definitely possible in Mapserver, I can provide a working example
they work when layer names have no spaces, but seem to fail when layer names contain a space, error message is as follows:

`

Parameter 'layers' contains unacceptable layer names. `

What I can't determine is which server is returning the error (my local or the remote)

tchaddad · 2016-02-26T23:25:41Z

I think the GetLegendGraphic conversation deserves it's own Issue. The GetMap original issue is clearly solved and appropriately closed.

jrahkonen · 2016-02-28T09:14:11Z

How about thinking for a while and making a list of all cases which may have the same issue?

GetMap
GetLegendGraphics
I do not know if these requests are cascaded to the remote WMS:
GetFeatureInfo
GetStyles

I suppose that GetMap is using %20 uniformly in all places of the connection string as it is defined in a mapfile at LAYER level, for example if connection contains "&STYLES=style with spaces in name" or vendor parameters with spaces.

A WFS feature type name with space is fundamentally an error but such names exist and it might be good to check the behaviour also with cascaded WFS layers

GetFeature
I do not know if DescribeFeatureType is cascaded to the remote server or not.

jmckenna added a commit to jmckenna/MapServer that referenced this issue Feb 18, 2016

modify encodeurl logic for spaces (issue MapServer#5232)

56d3631

tbonfort added a commit to MapServer/msautotest_DEPRECATED that referenced this issue Feb 24, 2016

update for +/%20 url encoding changes

c54b0b9

MapServer/MapServer#5232

tbonfort added a commit that referenced this issue Feb 24, 2016

Change url encoding of spaces from + to %20 (#5232)

b6fb921

tbonfort closed this as completed Feb 24, 2016

tbonfort added this to the 7.0.1 milestone Feb 24, 2016

tbonfort mentioned this issue Feb 24, 2016

modify encodeurl logic for spaces (issue #5232) #5246

Closed

akrherz mentioned this issue Feb 7, 2018

WMS error: Parameter 'layers' contains unacceptable layer names SciTools/cartopy#1020

Closed

geographika mentioned this issue Oct 10, 2019

Layer names with spaces in fail to retrieve GetLegendGraphic - mapserver 7.0.7 #5893

Open

MapServer as WMS client, spaces in layer names on upstream server not supported. #5232

MapServer as WMS client, spaces in layer names on upstream server not supported. #5232

Comments

bblammo commented Feb 4, 2016

Mapfile

jrahkonen commented Feb 4, 2016

jmckenna commented Feb 4, 2016

tchaddad commented Feb 5, 2016

bblammo commented Feb 5, 2016

jmckenna commented Feb 11, 2016

jmckenna commented Feb 19, 2016

tbonfort commented Feb 24, 2016

tbonfort commented Feb 24, 2016

jmckenna commented Feb 24, 2016

tbonfort commented Feb 24, 2016

jmckenna commented Feb 24, 2016

bblammo commented Feb 25, 2016

jmckenna commented Feb 25, 2016

bblammo commented Feb 25, 2016

jmckenna commented Feb 25, 2016

jmckenna commented Feb 25, 2016

jmckenna commented Feb 25, 2016

tchaddad commented Feb 25, 2016

jmckenna commented Feb 25, 2016

jmckenna commented Feb 25, 2016

jmckenna commented Feb 25, 2016

jmckenna commented Feb 25, 2016

jmckenna commented Feb 26, 2016

jmckenna commented Feb 26, 2016

jmckenna commented Feb 26, 2016

jmckenna commented Feb 26, 2016

tchaddad commented Feb 26, 2016

bblammo commented Feb 26, 2016

jmckenna commented Feb 26, 2016

tchaddad commented Feb 26, 2016

bblammo commented Feb 26, 2016

bblammo commented Feb 26, 2016

tchaddad commented Feb 26, 2016

jmckenna commented Feb 26, 2016

jmckenna commented Feb 26, 2016

tchaddad commented Feb 26, 2016

jmckenna commented Feb 26, 2016

tchaddad commented Feb 26, 2016

tchaddad commented Feb 26, 2016

jrahkonen commented Feb 28, 2016