WFS AND Filters with BBOX are broken #5352

Closed
geographika opened this Issue Nov 20, 2016 · 5 comments

Projects

None yet

2 participants

@geographika
Contributor
geographika commented Nov 20, 2016 edited

The following WFS filter containing a BBOX and a property filter:

<Filter>
   <AND>
      <PropertyIsEqualTo>
         <PropertyName>SCALERANK</PropertyName>
         <Literal>2</Literal>
      </PropertyIsEqualTo>
      <BBOX>
         <PropertyName>GEOM</PropertyName>
         <Envelope>
            <lowerCorner>52 -7</lowerCorner>
            <upperCorner>55 -6</upperCorner>
         </Envelope>
      </BBOX>
   </AND>
</Filter>

Logs the following:

FLTLayerApplyPlainFilterToLayer(): ( ([SCALERANK] =2) ANDintersects([shape],fromText('POLYGON ((-7.0000000000000000 52.0000000000000000, -7.0000000000000000 55.0000000000000000, -6.0000000000000000 55.0000000000000000, -6.0000000000000000 52.0000000000000000, -7.0000000000000000 52.0000000000000000))')) = TRUE) , rect=-7.000000,52.000000,-6.000000,55.000000

Note that the AND keyword is stuck to the intersects keyword. MapServer then seems to apply the spatial filter using the "rect" rather than the filter itself, and returns the correct results without error. The problem however is that this filter string is converted to native database strings when using other data sources, e.g for the MSSQL driver it throws the following error:

msMSSQL2008LayerTranslateFilter(): General error message. Translation to native SQL failed.

Tested using the following command:

mapserv -nh "QUERY_STRING=REQUEST=GetFeature&typeName=places&MAP=C:\mapfiles\wfs.map&SERVICE=WFS&VERSION=1.1.0&srsName=EPSG:4326&Filter=<Filter><AND><PropertyIsEqualTo><PropertyName>SCALERANK</PropertyName><Literal>2</Literal></PropertyIsEqualTo><BBOX><PropertyName>GEOM</PropertyName><Envelope><lowerCorner>52 -7</lowerCorner><upperCorner>55 -6</upperCorner></Envelope></BBOX></AND></Filter>"

Using Mapserver 7.0.2.

@geographika
Contributor

Sample MAP file:

MAP
    WEB
        METADATA
            "wfs_enable_request" "*"
            "wfs_onlineresource" "http://localhost/mapserver/?"            
        END
    END
    PROJECTION
        "init=epsg:4326"
    END

    EXTENT -180 -90 180 90    
    DEBUG 5
    CONFIG "MS_ERRORFILE" "C:\Temp\wfs.log"
    LAYER
        METADATA
            "gml_featureid"     "Name"
        END
    
        NAME 'places'
        TYPE POLYGON
        DATA '../data/vector/naturalearth/ne_110m_populated_places'
    END
END

Sample data at https://github.com/geographika/mapserver-tutorial-data/tree/master/data/vector/naturalearth (ne_110m_populated_places.shp)

Seems like the issue could be at: https://github.com/mapserver/mapserver/blob/66309eebb7ba0dc70469efeb40f865a8e88fafbd/mapogcfiltercommon.c#L382

The sprintf space created before adding the second filter clause is never used.

    pszExpression = msStringConcatenate(pszExpression, psFilterNode->pszValue);
    sprintf(szBuffer, "%s", " ");

    pszTmp = FLTGetCommonExpression(psFilterNode->psRightNode, lp);
    if (!pszTmp) {
      msFree(pszExpression);
      return NULL;
    }
	
    // could add in a space here?
    //sprintf(szBuffer, "%s", " ");
    //pszExpression = msStringConcatenate(pszExpression, szBuffer);

    pszExpression = msStringConcatenate(pszExpression, pszTmp);
    msFree(pszTmp);

There is no space for the intersects function in the same file:

sprintf(szBuffer, "%s", "intersects");

However many of the "psRightNode" expressions already have spaces built into them, resulting in 2 spaces, which whilst not a breaking issue makes the Filter look incorrect at a first glance.

sprintf(szBuffer, "%s", " AND ");

@rouault rouault added a commit to rouault/mapserver that referenced this issue Nov 25, 2016
@rouault rouault Filters: add whitespace between and and next part (relates to #5352)
A <Filter><And>....<BBOX></BBOX></And></Filter> results in
a ( ..... ANDintersects(....)) mapserver expression. The expression lexer seems
to make sense of it though, but this is ugly.
5b6ccd0
@rouault
Contributor
rouault commented Nov 25, 2016

I've committed a fix for those whitespace issues, but surprisingly the lexer seems to be able to separate ANDintersects correctly. So I'm not sure if your issue with MSSQL2008 is not something specific to it.

@geographika
Contributor
geographika commented Nov 26, 2016 edited

@rouault - thanks!

The lexer didn't throw an error without the space, but the spatial filter seemed to be applied using the rect part of the statement. I'll check again in the debugger.

FLTLayerApplyPlainFilterToLayer(): ( ([SCALERANK] =2) AND intersects([shape],fromText('POLYGON ((-7.0000000000000000 52.0000000000000000, -7.0000000000000000 55.0000000000000000, -6.0000000000000000 55.0000000000000000, -6.0000000000000000 52.0000000000000000, -7.0000000000000000 52.0000000000000000))')) = TRUE) , rect=-7.000000,52.000000,-6.000000,55.000000

I'll open a separate ticket for the MSSQL driver. The msMSSQL2008LayerTranslateFilter now crashes when it hits the comma after [shape]. I presume the [shape],fromText syntax is correct?

@rouault
Contributor
rouault commented Nov 26, 2016

From what I saw, the intersects() was correctly parsed as a MS_TOKEN_COMPARISON_INTERSECTS token, and it was not a rect statement. In the test I added, the filter was something like ( condition OR ( another_condition AND BBOX) ), and in that case, the BBOX cannot be applied as a rect since it's not a "top" BBOX.

@AlexanderGabriel AlexanderGabriel added a commit to AlexanderGabriel/mapserver that referenced this issue Nov 26, 2016
@rouault @AlexanderGabriel rouault + AlexanderGabriel Filters: add whitespace between and and next part (relates to #5352)
A <Filter><And>....<BBOX></BBOX></And></Filter> results in
a ( ..... ANDintersects(....)) mapserver expression. The expression lexer seems
to make sense of it though, but this is ugly.
dc67d28
@szekerest szekerest added a commit to szekerest/mapserver that referenced this issue Nov 27, 2016
@szekerest szekerest Reorder spatial operator arguments for mssql (#5352) 51415f7
@geographika
Contributor

@szekerest - many thanks for the driver fixes! The filter now translates perfectly in a MSSQL WHERE clause. My WFS layers are 50% quicker now the filtering is done in the database rather than in MapServer.

@rouault - I went through the debugger again, and read up on how the lexer works, and the lack of a space may have been a false issue. It certainly looked like it was an error in the logs, so the clean-up will definitely help someone in the future. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment