WFS AND Filters with BBOX are broken

[geographika]

The following WFS filter containing a BBOX and a property filter:

<Filter>
   <AND>
      <PropertyIsEqualTo>
         <PropertyName>SCALERANK</PropertyName>
         <Literal>2</Literal>
      </PropertyIsEqualTo>
      <BBOX>
         <PropertyName>GEOM</PropertyName>
         <Envelope>
            <lowerCorner>52 -7</lowerCorner>
            <upperCorner>55 -6</upperCorner>
         </Envelope>
      </BBOX>
   </AND>
</Filter>

Logs the following:

FLTLayerApplyPlainFilterToLayer(): ( ([SCALERANK] =2) ANDintersects([shape],fromText('POLYGON ((-7.0000000000000000 52.0000000000000000, -7.0000000000000000 55.0000000000000000, -6.0000000000000000 55.0000000000000000, -6.0000000000000000 52.0000000000000000, -7.0000000000000000 52.0000000000000000))')) = TRUE) , rect=-7.000000,52.000000,-6.000000,55.000000

Note that the AND keyword is stuck to the intersects keyword. MapServer then seems to apply the spatial filter using the "rect" rather than the filter itself, and returns the correct results without error. The problem however is that this filter string is converted to native database strings when using other data sources, e.g for the MSSQL driver it throws the following error:

msMSSQL2008LayerTranslateFilter(): General error message. Translation to native SQL failed.

Tested using the following command:

mapserv -nh "QUERY_STRING=REQUEST=GetFeature&typeName=places&MAP=C:\mapfiles\wfs.map&SERVICE=WFS&VERSION=1.1.0&srsName=EPSG:4326&Filter=<Filter><AND><PropertyIsEqualTo><PropertyName>SCALERANK</PropertyName><Literal>2</Literal></PropertyIsEqualTo><BBOX><PropertyName>GEOM</PropertyName><Envelope><lowerCorner>52 -7</lowerCorner><upperCorner>55 -6</upperCorner></Envelope></BBOX></AND></Filter>"

Using Mapserver 7.0.2.

[geographika]

Sample MAP file:

MAP
    WEB
        METADATA
            "wfs_enable_request" "*"
            "wfs_onlineresource" "http://localhost/mapserver/?"            
        END
    END
    PROJECTION
        "init=epsg:4326"
    END

    EXTENT -180 -90 180 90    
    DEBUG 5
    CONFIG "MS_ERRORFILE" "C:\Temp\wfs.log"
    LAYER
        METADATA
            "gml_featureid"     "Name"
        END
    
        NAME 'places'
        TYPE POLYGON
        DATA '../data/vector/naturalearth/ne_110m_populated_places'
    END
END

Sample data at https://github.com/geographika/mapserver-tutorial-data/tree/master/data/vector/naturalearth (ne_110m_populated_places.shp)

Seems like the issue could be at: https://github.com/mapserver/mapserver/blob/66309eebb7ba0dc70469efeb40f865a8e88fafbd/mapogcfiltercommon.c#L382

The sprintf space created before adding the second filter clause is never used.

    pszExpression = msStringConcatenate(pszExpression, psFilterNode->pszValue);
    sprintf(szBuffer, "%s", " ");

    pszTmp = FLTGetCommonExpression(psFilterNode->psRightNode, lp);
    if (!pszTmp) {
      msFree(pszExpression);
      return NULL;
    }
	
    // could add in a space here?
    //sprintf(szBuffer, "%s", " ");
    //pszExpression = msStringConcatenate(pszExpression, szBuffer);

    pszExpression = msStringConcatenate(pszExpression, pszTmp);
    msFree(pszTmp);

There is no space for the intersects function in the same file:

sprintf(szBuffer, "%s", "intersects");

However many of the "psRightNode" expressions already have spaces built into them, resulting in 2 spaces, which whilst not a breaking issue makes the Filter look incorrect at a first glance.

sprintf(szBuffer, "%s", " AND ");

[rouault]

I've committed a fix for those whitespace issues, but surprisingly the lexer seems to be able to separate ANDintersects correctly. So I'm not sure if your issue with MSSQL2008 is not something specific to it.

[geographika]

@rouault - thanks!

The lexer didn't throw an error without the space, but the spatial filter seemed to be applied using the rect part of the statement. I'll check again in the debugger.

FLTLayerApplyPlainFilterToLayer(): ( ([SCALERANK] =2) AND intersects([shape],fromText('POLYGON ((-7.0000000000000000 52.0000000000000000, -7.0000000000000000 55.0000000000000000, -6.0000000000000000 55.0000000000000000, -6.0000000000000000 52.0000000000000000, -7.0000000000000000 52.0000000000000000))')) = TRUE) , rect=-7.000000,52.000000,-6.000000,55.000000

I'll open a separate ticket for the MSSQL driver. The msMSSQL2008LayerTranslateFilter now crashes when it hits the comma after [shape]. I presume the [shape],fromText syntax is correct?

[rouault]

From what I saw, the intersects() was correctly parsed as a MS_TOKEN_COMPARISON_INTERSECTS token, and it was not a rect statement. In the test I added, the filter was something like ( condition OR ( another_condition AND BBOX) ), and in that case, the BBOX cannot be applied as a rect since it's not a "top" BBOX.

[geographika]

@szekerest - many thanks for the driver fixes! The filter now translates perfectly in a MSSQL WHERE clause. My WFS layers are 50% quicker now the filtering is done in the database rather than in MapServer.

@rouault - I went through the debugger again, and read up on how the lexer works, and the lack of a space may have been a false issue. It certainly looked like it was an error in the logs, so the clean-up will definitely help someone in the future. Thanks.