SLD with FILTER doesn't apply properly to POSTGIS LAYERS

[mapserver-bot]

Reporter: guillaume
Date: 2011/12/06 - 12:19
Trac URL: http://trac.osgeo.org/mapserver/ticket/4112
Using MapServer 6 and SLD gives incorrect results for POSTGIS LAYERS (and maybe others)

• SLD filter is used to build a FILTER into the LAYER block, but no EXPRESSION is written into the CLASSes blocks. As a result, all records gets first CLASS representation on the final map.
  MapServer5 is OK
  MapServer6 is OK with a SHAPEFILE data (same SLD used). In this case, no FILTER is written and the EXPRESSION statements are correctly put into CLASSes definitions.

[mapserver-bot]

Author: havatv
Date: 2012/01/06 - 01:13
I have observed the same problems. SLD that worked for PostGIS layers in 5 does not work anymore in 6.

[mapserver-bot]

Author: pgoulborn
Date: 2012/01/23 - 13:06
We're being bitten by this issue too, to the point where we're having to stick with 5.6 until it gets resolved. One part of each of the filters for a layer is being written into the FILTER and is being taken out of/not written into the CLASS EXPRESSIONS when the layer is using POSTGIS.

In our case the symptoms are that when we pass a custom SLD to a LAYER backed by PostGIS all features are styled with the first style defined for the layer. Using the same SLD with Mapserver 5.6 on the same server works as expected.

Casting a very inexpert eye over the source it seems as though there is code to create the FILTER for layers fronting database data only (source:trunk/mapserver/mapogcsld.c@12904#L431) and I would guess that's where the issue arises.

Is there any way to expedite the investigation/resolution of the problem, through a bounty or directly funding development?

[mapserver-bot]

Author: dmorissette
Date: 2012/01/23 - 15:03
See also ticket #4152 which seems to be a report of the same issue.

[mapserver-bot]

Author: guillaume
Date: 2012/01/24 - 16:02
I've found where the issue was. On lines 433-440 in mapogcsld.c (file from version 6.0.1, as it has changed a bit in trunk already) :

    if (!bFailedExpression)
    {
        snprintf(szTmp, sizeof(szTmp), "%s", "))");
        pszBuffer =msStringConcatenate(pszBuffer, szTmp);
        msLoadExpressionString(&lp->filter, pszBuffer);
        /* '''HERE IT IS. SIMPLY UNCOMMENT THESE TWO LINES'''
            for (k=0;k<lp->numclasses;k++)
                freeExpression(&lp->class[k]->expression);
        */
    }

Actually I'm always reluctant to remove/comment something which seems to have been done by purpose, but it definitely fixes the issue.

[mapserver-bot]

Author: aboudreault
Date: 2012/01/24 - 16:25
Thanks guillaume for your comment. I started looking into this as well and have a potential fix in mind that I will commit in the next day or two once we have done more testing on it. I was working with the ticket 4152 test case and I found a completely different issue, that fixed the ticket properly. I'll need to review all this and ensure we have a solution that fixes both tickets.

[mapserver-bot]

Author: aboudreault
Date: 2012/01/26 - 20:14
I've committed a fix for BRANCH 6.0 in 9b1bca2 (r13014). Basically, there was 2 bugs since the sld can passes through a different part of the code.

• Bug 1: When the SLD ''does not'' contain any AND/OR operator in the rules, the FILTER is properly generated but the expression are not written in the mapfile... the style is invalid. After a few tests, the Guillaume patch is ok.
• Bug 2: When the SLD ''does'' contain AND/OR operators, the filter is incorrectly generated. Only the first rule is used.

My changeset fixes both issues. However, nothing seems to work in trunk and I'll need to debug this. The trunk code changed and the behavior is different and broken.

[mapserver-bot]

Author: pgoulborn
Date: 2012/01/27 - 15:54
I've got a build of 9b1bca2 (r13014) and it's looking good so far - the cases where we hit the error before are now returning identical images to 5.6.6

[mapserver-bot]

Author: pgoulborn
Date: 2012/01/27 - 18:11
Hmm... I've just hit an issue with FILTER in WFS GetFeature requests - I can't say whether this is a regression since 6.0.1 release but the same filters work with 5.6.6.

From the logs it looks as though a temporary map file should be created with a FILTER for the LAYER - but no FILTER is present.

[mapserver-bot]

Author: aboudreault
Date: 2012/01/27 - 20:55
pgoulborn, could you create a simple test case, or at least provide me the SLD + a mapserver URL example you are using. This might be a new issue, not necessary related to this one. If so, we'll create another ticket.

[mapserver-bot]

Author: pgoulborn
Date: 2012/01/30 - 10:25
You're right, it doesn't appear to be related and I've tested against 6.0.1 and get the same result. I'm a bit tied up at the moment so I'm going to go back to 5.6 for now, but I'll open a new ticket once I have more information or something you can test against.

As far as I'm concerned, the problem in this ticket is resolved in 6.0 9b1bca2 (r13014)

[mapserver-bot]

Author: pgoulborn
Date: 2012/02/07 - 11:46
I finally had some more time to look at the not-related-to-this-case FILTER issue and the problem was with me and not Mapserver, I'm posting here in case anyone else encounters the same issue.

I was using an incorrect filter: <Intersect> rather than <Intersects>. This works with 5.6 but not 6.0, I was being lazy and picked an example of the filter from http://mapserver.org/ogc/filter_encoding.html#tests which, despite elsewhere on the page talking about "Intersects", uses "Intersect" in the actual test. I guess that page is out of date - and the demo server is as well (GetCapabilities says 5.6.5)?

[mapserver-bot]

Author: havatv
Date: 2012/02/07 - 12:41
Replying to [comment:13 pgoulborn]:

I was using an incorrect filter: rather than . This works with 5.6 but not 6.0, I was being lazy and picked an example of the filter from http://mapserver.org/ogc/filter_encoding.html#tests which, despite elsewhere on the page talking about "Intersects", uses "Intersect" in the actual test. I guess that page is out of date - and the demo server is as well (GetCapabilities says 5.6.5)?

I have changed Intersect to Intersects
for the filter encoding page (MapServer/MapServer-documentation@a4a211c (r13077)).

[mapserver-bot]

Author: aboudreault
Date: 2012/02/08 - 19:29
After another fix related to SLD 355f0a9 (r13021), Initial issue is now fixed in trunk in 8705658 (r13085).

[mapserver-bot]

Author: guillaume
Date: 2012/03/05 - 16:24
Hi,
If that issue is closed, we have still a problem with FILTER. When a FILTER clause exists on a LAYER and you apply an SLD file (without any filter) to it, the FILTER CLAUSE is not removed and thus only the objects that pass the filter are displayed.

[mapserver-bot]

Author: havatv
Date: 2012/03/05 - 16:38
Replying to [comment:16 guillaume]:

Hi,
If that issue is closed, we have still a problem with FILTER. When a FILTER clause exists on a LAYER and you apply an SLD file (without any filter) to it, the FILTER CLAUSE is not removed and thus only the objects that pass the filter are displayed.

I don't understand why you would want layer filters to be removed in the presence external styling / SLD. I would definitely want my layer filters to be enforced, as they have nothing to do with the styling (they only define which features I want to be make available in the layer).

[mapserver-bot]

Author: guillaume
Date: 2012/03/05 - 16:44
Replying to [comment:17 havatv]:

Replying to [comment:16 guillaume]:

Hi,
If that issue is closed, we have still a problem with FILTER. When a FILTER clause exists on a LAYER and you apply an SLD file (without any filter) to it, the FILTER CLAUSE is not removed and thus only the objects that pass the filter are displayed.

I don't understand why you would want layer filters to be removed in the presence external styling / SLD. I would definitely want my layer filters to be enforced, as they have nothing to do with the styling (they only define which features I want to be make available in the layer).

As seen previously, MapServer applySLD adds a FILTER clause to any POSTGIS LAYER when an SLD filter must be applied. So after applying an SLD containing a filter in it, you end up with a LAYER containing a FILTER. If you want to remove that filter, to display all the data like it was before SLD application, you can't, because even if you apply an SLD file without any filter in it, your layer's FILTER is not removed from the mapfile.

[mapserver-bot]

Author: dmorissette
Date: 2012/03/06 - 06:13
You talk about restoring state after applying a SLD, so I presume you are not using the straight CGI, but instead are using some kind of long-running process (FastCGI or MapScript)? Sounds like this is a different issue from this ticket's original problem and should get its own ticket with more details on your setup and/or what you are doing exactly.

[mapserver-bot]

Author: guillaume
Date: 2012/03/06 - 09:19
Replying to [comment:19 dmorissette]:

You talk about restoring state after applying a SLD, so I presume you are not using the straight CGI, but instead are using some kind of long-running process (FastCGI or MapScript)? Sounds like this is a different issue from this ticket's original problem and should get its own ticket with more details on your setup and/or what you are doing exactly.

Hi Daniel,
Actually I'm just saving map files once modified with applySLD method. And it's then impossible to remove the FILTER clause. But as havatv said, a preset FILTER shouldn't be removed either. That's why I guess this "add filter when applying SLD on DB layers" optimization is not a good idea. If the FILTER must be protected when applying an SLD file, why could it be set during the same operation ? This FILTER thing as lead to many bogus behaviors, like the one described sooner in this report. SLD is about styling, and should only impact the CLASS/STYLES blocks, not the LAYER level. If any optimization can be done using filter, it should be done deeper in mapserver core, when mapserver reads the layer definition and sees CLASSES are using expressions.
My 2 cents.

Guillaume

[mapserver-bot]

attachment http://trac.osgeo.org/mapserver/attachment/ticket/4112/mapserv5.png :

   result with MS5

[mapserver-bot]

attachment http://trac.osgeo.org/mapserver/attachment/ticket/4112/mapserv6.png :

   result with MS6

[mapserver-bot]

attachment http://trac.osgeo.org/mapserver/attachment/ticket/4112/test.sld :

   sld file used

[Sverre-Iversen]

This must be related to #4363 as well I guess? Can you read it and comment on it?

[tbonfort]

cc @aboudreault @dmorissette @havatv @gsueur (pinging you all since you've not been notified of the update after the github migration)

[tbonfort]

previous commit was erroneously attributed to 4112, whereas it should have referenced #4412

[tbonfort]

considering this one closed in 6.2, c.f. https://github.com/mapserver/msautotest/blob/master/wxs/wms_sld_postgis.map