RFC 118 & Geotransform

[pelord]

Here my class for a linestring stored in postgis as a multilinestring.

CLASSITEM "state"
CLASS
	NAME "bad"
	EXPRESSION 'bad'
	STYLE
		LINECAP butt
		WIDTH 8
		COLOR 255 165 0
	END
	STYLE
		GEOMTRANSFORM "end"
		SYMBOL "vertline"
		COLOR 0 0 0
		SIZE 13
		WIDTH 1
		ANGLE AUTO
	END
	STYLE
		GEOMTRANSFORM "start"
		SYMBOL "vertline"
		COLOR 0 0 0
		SIZE 13
		WIDTH 1
		ANGLE AUTO
	END
END
CLASS
...
END 

If I don't apply a wms filter, everything is rendered (getmap) as in the class's definition.

If a apply a filter (rfc 118):

<Filter xmlns="http://www.opengis.net/ogc"><PropertyIsEqualTo matchCase="true"><PropertyName>state</PropertyName><Literal>bad</Literal></PropertyIsEqualTo>
Here the result.

A query on wms with geotransform end/start seem to render a densified version of the linestring.

Here the vertices of this line:

I've tried to convert the multilinestring to a linestring as done on gis.stackexchange with the same result.

Is it an issue or a bad mapfile definition?

MapServer version 7.2.1

[pelord]

Same result with GEOMTRANSFORM "vertices".

[sdlime]

I think it's a bug, likely in the query map code. What happens if you add:

QUERYMAP
  STYLE NORMAL
END

to your mapfile?

[pelord]

Same result.

[sdlime]

Then it's a problem in the msDrawQuery() function in mapdraw.c not properly handling GEOMTRANSFORMs... nuts. Dan alluded to other issues in the limitations section of the RFC and I guess this is another one. I'm curious if POLYGONs behave the same. The function in question uses msDrawShape() for polygons but calls lower-level functions for lines.

[pelord]

Polygons do not have the same behavior (geotransform "start")

Without filter:

With filter:

[sdlime]

That's what I was hoping would happen - it's a good hint.. The question is if msDrawShape() can also be used for line features in msDrawQuery().

[pelord]

I can't answer to this question :)

[sdlime]

@pelord, what does your vertline symbol definition look like?

[pelord]

	SYMBOL
		NAME "vertline"
		TYPE vector
		FILLED true
		POINTS
			0 0
			0 5
			1 5
			1 0
			0 0
		END
	END

[pelord]

@geographika Do you think this could be related to your issue?

http://osgeo-org.1560.x6.nabble.com/Combining-WMS-Filter-and-Style-Parameters-td5411882.html

[geographika]

@pelord - yes it looks like it is the same issue, although I'm not using GEOTRANSFORM - just filters and styles on a WMS.
I've made a simple test case to demonstrate the issue (tested on MapServer 7.4.1 on Windows) which may help @sdlime to understand what is going on. @dmorissette any idea if this a known limitiation of RFC118 or a bug?

The example uses test_wms_filter.map.txt (remove the .txt extension required to upload to GitHub).

A basic WMS request with the &STYLES=green style selected:

mapserv -nh "QUERY_STRING=map=test_wms_filter.map&layers=INLINE&SERVICE=WMS&REQUEST=GetMap&VERSION=1.3.0&STYLES=green&FORMAT=image/png&TRANSPARENT=true&HEIGHT=256&WIDTH=256&CRS=epsg:4326&BBOX=-10,-10,10,10" > test_green.png

Works fine:

A basic WMS request with the &STYLES=red:

mapserv -nh "QUERY_STRING=map=test_wms_filter.map&layers=INLINE&SERVICE=WMS&REQUEST=GetMap&VERSION=1.3.0&STYLES=red&FORMAT=image/png&TRANSPARENT=true&HEIGHT=256&WIDTH=256&CRS=epsg:4326&BBOX=-10,-10,10,10" > test_red.png

The failing case is below - &STYLES=red but with a &filter=<Filter</Filter> clause. Records are filterered correctly but displayed in green. It seems that whatever is set in CLASSGROUP is used for the style not the one passed in by the WMS. If CLASSGROUP is removed then it displays whichever CLASS is first for the layer.

mapserv -nh "QUERY_STRING=map=test_wms_filter.map&layers=INLINE&SERVICE=WMS&REQUEST=GetMap&VERSION=1.3.0&STYLES=red&FORMAT=image/png&TRANSPARENT=true&HEIGHT=256&WIDTH=256&CRS=epsg:4326&BBOX=-10,-10,10,10&filter=<Filter><PropertyIsEqualTo><PropertyName>property1</PropertyName><Literal>value1</Literal></PropertyIsEqualTo></Filter>" > test_with_filter.png

[pelord]

Let me know if I could help!

[geographika]

I've done some debugging, and I think I know why this is happening, but not much idea how to fix it.

I believe it is the same issue as referenced in the RFC:

Since rendering of the filtered layers is done via the QUERYMAP mechanism, the MASK, OPACITY and COMPOSITE settings are ignored since they are not implemented for QUERYMAPs. (This could be considered a QUERYMAP bug but is out of scope for this RFC)

I believe this is a similar case for CLASSGROUP (and possibly GEOTRANSFORM).

A layer with a filter uses msDrawQueryLayer rather than the standard msDrawLayer - decided in mapdraw.c

There is a comment in msDrawQueryLayer:

** Layer was opened as part of the query process, msLayerWhichItems() has also been run, shapes have been classified - start processing!

msLayerWhichItems is called earlier - but the classgroup is set to 0 at this stage, so layer->resultcache may not have taken into account the CLASSGROUP set by the style?

[sdlime]

I know where in the code the problem lies. What’s not clear is how to fix it. The query layer rendering probably needs to be refactored so that it uses the regular pipeline as much as possible. This isn’t specific to WMS. Straight MapServer queries with an image qformat also demonstrate the issue.

…

________________________________ From: Seth G <notifications@github.com> Sent: Friday, August 2, 2019 2:52:00 AM To: mapserver/mapserver <mapserver@noreply.github.com> Cc: Lime, Steve D (MNIT) <steve.lime@state.mn.us>; Mention <mention@noreply.github.com> Subject: Re: [mapserver/mapserver] RFC 118 & Geotransform (#5744) This message may be from an external email source. Do not select links or open attachments unless verified. Report all suspicious emails to Minnesota IT Services Security Operations Center. I've done some debugging, and I think I know why this is happening, but not much idea how to fix it. I believe it is the same issue as referenced in the RFC<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmapserver.org%2Fdevelopment%2Frfc%2Fms-rfc-118.html&data=02%7C01%7Csteve.lime%40state.mn.us%7C23cc636373c74985ca2c08d7171e4d8f%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C637003291230624207&sdata=Uox%2FyF4jVEMGWQPFWsMcThWHR7k%2BbS%2FhyAd%2BMnyovZU%3D&reserved=0>: Since rendering of the filtered layers is done via the QUERYMAP mechanism, the MASK, OPACITY and COMPOSITE settings are ignored since they are not implemented for QUERYMAPs. (This could be considered a QUERYMAP bug but is out of scope for this RFC) I believe this is a similar case for CLASSGROUP (and possibly GEOTRANSFORM). A layer with a filter uses msDrawQueryLayer rather than the standard msDrawLayer - decided in mapdraw.c<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmapserver%2Fmapserver%2Fblob%2Fd2e05178ae54165481c47f2adc93c619c901bdae%2Fmapdraw.c%23L394&data=02%7C01%7Csteve.lime%40state.mn.us%7C23cc636373c74985ca2c08d7171e4d8f%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C637003291230634199&sdata=TtuMFvAG6vJO6nfWIHFTc9UX3wICUc%2BCh3d7E1JGQjQ%3D&reserved=0> There is a comment<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmapserver%2Fmapserver%2Fblob%2Fd2e05178ae54165481c47f2adc93c619c901bdae%2Fmapdraw.c%23L1402&data=02%7C01%7Csteve.lime%40state.mn.us%7C23cc636373c74985ca2c08d7171e4d8f%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C637003291230634199&sdata=4Iwe2O2l9VVmPpBhK5yF5dwRNQv4fm5bltxCbpnU%2FRU%3D&reserved=0> in msDrawQueryLayer: ** Layer was opened as part of the query process, msLayerWhichItems() has also been run, shapes have been classified - start processing! msLayerWhichItems is called earlier - but the classgroup is set to 0 at this stage, so layer->resultcache may not have taken into account the CLASSGROUP set by the style? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmapserver%2Fmapserver%2Fissues%2F5744%3Femail_source%3Dnotifications%26email_token%3DAAMGN62KI4GR3L5WUPFN2TLQCPRSBA5CNFSM4GYE5B62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3M6VSA%23issuecomment-517597896&data=02%7C01%7Csteve.lime%40state.mn.us%7C23cc636373c74985ca2c08d7171e4d8f%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C637003291230634199&sdata=kuWVwyW1%2BSfP3a4WYp6ph3nLMYSNSWnt0LFE8UdJ4IQ%3D&reserved=0>, or mute the thread<https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAMGN65TWK6EBDPCVTUYP7TQCPRSBANCNFSM4GYE5B6Q&data=02%7C01%7Csteve.lime%40state.mn.us%7C23cc636373c74985ca2c08d7171e4d8f%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C637003291230644188&sdata=TMay8G8WATp41C1cquON649k0NX4RdoCIUlB%2BUxgiKQ%3D&reserved=0>.

[geographika]

@sdlime - I'd already done some tests and debugging before @pelord pointed me to this issue so thought I'd add them in. Let me know if you need any testing done or test cases. The refactoring of msDrawLayer and msDrawQueryLayer looks challenging.

[geographika]

A minor update, I tried setting querymap to 0 here to force the "regular drawing pipeline". In the test case with the red and green points then nothing is displayed with msDrawLayer when using a filter.

In a much more complex system with a MSSQL based layer it does seem to resolve my issue, and display filtered features with the correct style.

Not sure if that provides anything useful.

[sdlime]

Can you guys give the pull request a test? There are likely still issues with HILITE querymap styles and knowing which defined style to change colors on in certain cases. However, this should address cases where a style sets an outline width and/or uses GEOMTRANSFORM at the style level.

[pelord]

Sorry, I do not have the experience/environnement for compiling Mapserver and test it. I'll try soon to do it but it might be extremely time consuming for me.

[geographika]

@sdlime - I've compiled the code from the issue-5744 branch (just for info this is about 6 months of commits behind master).

Unfortunately I still have the same issue where applying a filter reverts to using the first style - the third test here.

[sdlime]

Anyway to create a succinct version of the 3rd test?

[geographika]

@sdlime - I can add as an msautotest to your branch?

[sdlime]

No, that's ok, I now see the mapfile and sample calls above and can work with that. I don't think this is the same issue. @pelord's problems were specific to lines and the msDrawQueryLayer() code was clearly not following the same approach as msDrawVectorLayer() in terms of dealing with outlinewidth, GEOMTRANFORMed styles, etc... That's what the pull request fixes.

Your issue is with point layers which would have been drawn correctly as the code sits now. The msDrawQueryLayer() code doesn't assign a class index to a selected feature. That's done upstream in the query code (since that's where template references could possibly live). The drawing code just leverages the set class index and then tweaks colors if necessary (TYPE HILITE). Seems to me like the class index isn't being set properly elsewhere. That will be easy to verify. I know the main query functions respect the CLASSGROUP but I don't know what the WMS code is doing.

[sdlime]

I can confirm that the class index in the 3rd case is reported as 0, so CLASSGROUP "green". The problem is elsewhere in the query code I think.

I can further confirm that when msQueryByFilter() is called, layer INLINE has class group set to "green", so the default value but when msDrawQueryLayer() is called, layer INLINE has class group set to "red". The problem is in the WMS code, it's setting class group after the filtering is done and not before.

[sdlime]

@dmorissette, you're way more familiar with the WMS code than I am - only the previous comment is really relevant. Does the issue of setting a layers class group after doing a query seem like something that could be happening? I think changing where the query is executed in msWMSLoadGetMapParams() would do it - or moving the query into msWMSGetMap(),,,

[sdlime]

This has been fixed in both master and branch-7-4. Closing... --Steve

[pelord]

I am currently testing version 7.6.2 and I confirm that this problem is fixed. Sorry for the delay!

[sdlime]

Thanks @pelord!