Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

msGEOSShapeFromWKT(): GEOS library error. Error reading WKT geometry "". on request with geo filter. #6176

Closed
autra opened this issue Oct 26, 2020 · 19 comments
Assignees
Labels
Milestone

Comments

@autra
Copy link

autra commented Oct 26, 2020

Description

Mapserver somehow fails to call libgeos with a correct wkt, and then tries an invalid postgis request.

Steps to reproduce

  • send the following request:
curl --location --request POST 'https://wfs.environnement.brussels/test' \
--header 'Content-Type: text/plain' \
--data-raw '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<wfs:GetFeature maxFeatures="5000" version="1.1.0" service="WFS" xsi:schemaLocation="http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.1.0/wfs.xsd" xmlns:wfs="http://www.opengis.net/wfs" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <wfs:Query typeName="BELB:water_protectedsite_drinking_water">
        <wfs:PropertyName>gid</wfs:PropertyName>
        <wfs:PropertyName>msGeometry</wfs:PropertyName>
        <ogc:Filter>
            <ogc:Intersects>
                <ogc:PropertyName>msGeometry</ogc:PropertyName>
                <gml:MultiPolygon>
                    <gml:Polygon>
                        <gml:exterior>
                            <gml:LinearRing>
                                <gml:posList>149688.0 170683.0 149732.0 170669.0 149743.0 170699.0 149745.0 170706.0 149735.0 170709.0 149734.0 170706.0 149709.0 170714.0 149710.0 170717.0 149700.0 170721.0 149698.0 170714.0 149697.0 170714.0 149688.0 170683.0 </gml:posList>
                            </gml:LinearRing>
                        </gml:exterior>
                    </gml:Polygon>
                    <gml:Polygon>
                        <gml:exterior>
                            <gml:LinearRing>
                                <gml:posList>145004.0 169690.0 145005.0 169681.0 145006.0 169680.0 145007.0 169667.0 145012.0 169670.0 145011.0 169677.0 145011.0 169681.0 145009.0 169690.0 145004.0 169690.0 </gml:posList>
                            </gml:LinearRing>
                        </gml:exterior>
                    </gml:Polygon>
                </gml:MultiPolygon>
            </ogc:Intersects>
        </ogc:Filter>           
    </wfs:Query>
</wfs:GetFeature>'

  • The server answers with this:
<?xml version="1.0" encoding="UTF-8"?>
<ows:ExceptionReport xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows" version="1.1.0" language="en-US" xsi:schemaLocation="http://www.opengis.net/ows http://schemas.opengis.net/ows/1.0.0/owsExceptionReport.xsd">
  <ows:Exception exceptionCode="NoApplicableCode" locator="mapserv">
    <ows:ExceptionText>msWFSGetFeature(): WFS server error. FLTApplyFilterToLayer() failed
msPostGISLayerWhichShapes(): Query error. Error executing query. Check server logs
msTokenizeExpression(): Expression parser error. Parsing fromText function failed, WKT processing failed.
msGEOSShapeFromWKT(): GEOS library error. Error reading WKT geometry "".
msGEOSError(): GEOS library error.</ows:ExceptionText>
  </ows:Exception>
</ows:ExceptionReport>

Additional info

I think that for some reason, the wkt is not passed correctly to msGEOSShapeFromWKT. And then mapserver still tries to make a postgis request but fails with these error logs:

[Mon Oct 26 16:38:28 2020].439494 msGEOSError(): GEOS library error.
[Mon Oct 26 16:38:28 2020].439547 msGEOSShapeFromWKT(): GEOS library error. Error reading WKT geometry "".
[Mon Oct 26 16:38:28 2020].439554 msTokenizeExpression(): Expression parser error. Parsing fromText function failed, WKT processing failed.
[Mon Oct 26 16:38:28 2020].439759 msPostGISLayerWhichShapes(): Error (ERROR:  syntax error at or near ")"
LINE 1: ...21794 0,0 0))',31370) and (st_intersects(the_geom,) limit 50...
                                                             ^
) executing query: select "gid"::text,"zone_type"::text,ST_AsBinary(ST_Force2D("the_geom"),'NDR') as geom,"gid"::text from web_service.wsl_bruenvi_water_protectedsite_drinking_water where "the_geom" && ST_GeomFromText('POLYGON((0 0,0 3104546.30092,3937643.21794 3104546.30092,3937643.21794 0,0 0))',31370) and (st_intersects(the_geom,) limit 5001
[Mon Oct 26 16:38:28 2020].439767 msPostGISLayerWhichShapes(): Query error. Error executing query. Check server logs
[Mon Oct 26 16:38:28 2020].439772 msWFSGetFeature(): WFS server error. FLTApplyFilterToLayer() failed

I don't know if the first error is normal or not, but the second tends to show at least that mapserver didn't recover well from the first one.

This is a test url and I can compile and install a new mapserver version if needed.

Context

Server: Debian Stretch
mapserver: v7.6.0
libgeos: 3.7.1 (but I reproduced with 3.5 as well)

Thanks!

@autra
Copy link
Author

autra commented Jan 25, 2021

I'm still interested in seeing this fixed. What can I provide to help here?

@jmckenna
Copy link
Member

When you report a problem here, please always take the time to create a small test case (mapfile, data, commands.txt) and attach it to your ticket as a .zip.

@geographika
Copy link
Member

@autra a few steps to narrow it down:

  1. Does the request work if you don't use a MultiPolygon? Does it work with a simple polygon?
  2. Does the request work on another layer in MapServer?
  3. Does it work with VERSION=1.0.0? See examples at https://www.mapserver.org/ogc/filter_encoding.html#tests
  4. Can you check the GML is valid? Maybe with OGR?

@autra
Copy link
Author

autra commented Feb 2, 2021

I'm working on a test case. Meanwhile

  1. It does work with a simple polygon
  2. It fails with every layer I've tested (3 different)
  3. Tested with VERSION=1.0.0, it does not work
  4. The gml is valid. QGis server answers correctly with this query btw.

Thanks

@sdlime
Copy link
Member

sdlime commented Feb 26, 2021

Well, I suppose we need to intercept the WKT being generated by the WFS code and figure out where the WFS filter GML to WKT conversion is screwed up. @autra, any luck with the test case?

@sdlime sdlime added this to the 8.0 Release milestone Feb 26, 2021
@autra
Copy link
Author

autra commented Feb 26, 2021

Didn't get the bandwidth to finish it, but it's in progress :-)

@autra
Copy link
Author

autra commented Mar 12, 2021

I've attached a test case as minimal as I could here: test_case.zip. Thanks!

@jmckenna
Copy link
Member

jmckenna commented Mar 12, 2021

@autra that is sure an intense test case (I feel like a paid client just sent me this dump and I must spend an entire day to understand the data ha), I can imagine not many will take the time to understand your db layout like I tried (multiple schemas etc.). Your mapfile references a 'web_service' schema, but there are no tables inside that schema in your dump, I only see a 'i_pa_capt_zone2et3_area_f' table in the 'water' schema.

How about this plan, can you just create one public table (no special schemas) in a new database (preferably a SpatiaLITE db, so everyone here can reproduce your problem without installing PostgreSQL/PostGIS), that causes your error, and include it in a zip archive with a mapfile, commands.txt, and your SpatiaLITE db file?

@autra
Copy link
Author

autra commented Mar 12, 2021

@jmckenna

that is sure an intense test case (I feel like a paid client just sent me this dump and I must spend an entire day to understand the data ha)

I'm not sure why you say that. It's a bit harsh to me given the fact that I've already make it a lot simpler that it was from the beginning. Beside this db is pretty simple, it has only one table and one view (in web_service, it's not a table). pg_restore --list test_case.dump gives you a list of objects. I'm sorry that this is not enough for you, but for me it isn't really complicated or anything.

But whatever, let's move on: here is another test case, with one table: public.test_case, with one row, which reproduces the problem:
test_case.zip

I hope it's simple enough this time.

preferably a SpatiaLITE db

Sorry but no. I've never used spatialite db, and I'm not going to learn another sgbd, adapt my data, without even the certainty to reproduce. I understand all about maintainer fatigue (I really do!) but I just can't do that and as mapserver supports postgis, I feel like test cases with PostGIS should be accepted.

Thanks.

@jratike80
Copy link

SQL dump as text might be even more simple https://gdal.org/drivers/vector/pgdump.html

If you don't know how to do something you can ask for help. For example ogr2ogr is pretty good in converting data, not tested but probably close to working command for creating the SQLite database:

ogr2ogr -f SQLite -dsco spatialite=yes output.sqlite PG:"host=localhost port=5432 dbname=my_db user=user password=password" -sql "SELECT * from my_table limit 10" -nln my_table

I know I could construct the WKT of the multipolygon from your filter but I also feel a bit lazy. Could you add it? I guess it is valid if QGIS accepts the filter but still.

@jmckenna
Copy link
Member

Thanks @autra the first test case failed with MapServer. For the second test case, I had to change the extents to the extent of the geometry, and also added a CLASS object so it displays in MapServer. I can then get a valid map image returned from MapServer. Will try a GetFeature WFS request now...

@jratike80
Copy link

jratike80 commented Mar 12, 2021

Well, I did it anyway. It is a valid multipolygon, I guess it presents two building footprints which are about 4.8 km apart.

MULTIPOLYGON ((( 149688 170683, 149732 170669, 149743 170699, 149745 170706, 149735 170709, 149734 170706, 149709 170714, 149710 170717, 149700 170721, 149698 170714, 149697 170714, 149688 170683 )), (( 145004 169690, 145005 169681, 145006 169680, 145007 169667, 145012 169670, 145011 169677, 145011 169681, 145009 169690, 145004 169690 )))

It seems that filter goes beyond the extent of the layer. I am not sure if it matters.

LAYER
        NAME "water_protectedsite_drinking_water"
        EXTENT 149729.1576 161464.6568 152875.1035 167345.1646

@jmckenna
Copy link
Member

To Jukka's point, the more time you spend on your side making it easy to reproduce (learning the one ogr2ogr command to convert to Spatialite, and testing that, then packaging it), the more chances that you'll get a developer's free time to help you. I see that you feel that it is my job to figure out your data schema, that's an interesting view (pun intended). Ok back to testing your GetFeature request here...

@jmckenna
Copy link
Member

My single polygon intersect filter works fine:

mapserv -nh QUERY_STRING="MAP=/ms4w/apps/ticket-6176-geos-wkt/mapfile.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=test_case&FILTER=<Filter><Intersects><PropertyName>msGeometry</PropertyName><gml:Polygon><gml:outerBoundaryIs><gml:LinearRing><gml:coordinates>150529.864814123 165991.022270877,151356.909753477 165715.340624425,151414.343429821 164784.915067652,151310.962812402 164199.091568943,150977.847489607 164199.091568943,150817.033195843 164819.375273459,151230.55566552 165106.543655179,150771.086254768 165301.818154748,150346.077049822 165497.092654318,150346.077049822 165497.092654318,150529.864814123 165991.022270877</gml:coordinates></gml:LinearRing></gml:outerBoundaryIs></gml:Polygon></Intersects></Filter>"

results:

<?xml version='1.0' encoding="UTF-8" ?>
<wfs:FeatureCollection
   xmlns:TEST="http://mapserver.gis.umn.edu/mapserver"
   xmlns:wfs="http://www.opengis.net/wfs"
   xmlns:gml="http://www.opengis.net/gml"
   xmlns:ogc="http://www.opengis.net/ogc"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.0.0/WFS-basic.xsd
                       http://mapserver.gis.umn.edu/mapserver http://127.0.0.1/cgi-bin/mapserv.exe?MAP=/ms4w/apps/ticket-6176-geos-wkt/mapfile.map&amp;SERVICE=WFS&amp;VERSION=1.0.0&amp;REQUEST=DescribeFeatureType&amp;TYPENAME=test_case&amp;OUTPUTFORMAT=XMLSCHEMA">
      <gml:boundedBy>
        <gml:Box srsName="EPSG:31370">
                <gml:coordinates>149729.157600,161464.656800 152875.103500,167345.164600</gml:coordinates>
        </gml:Box>
      </gml:boundedBy>
    <gml:featureMember>
      <TEST:test_case fid="test_case.1">
        <gml:boundedBy>
                <gml:Box srsName="EPSG:31370">
                        <gml:coordinates>149729.157600,161464.656800 152875.103500,167345.164600</gml:coordinates>
                </gml:Box>
        </gml:boundedBy>
        <TEST:msGeometry>
        <gml:Polygon srsName="EPSG:31370">
          <gml:outerBoundaryIs>
            <gml:LinearRing>
              <gml:coordinates>150738.025000,166844.402500 150782.650000,166797.762500 150819.915000,166763.348500 150868.821000,166724.230500 150898.946000,166697.550500 150939.150000,166657.230500 150991.056000,166593.465500 151014.134000,166559.871500 151040.486000,166515.113500 151069.142000,166459.926500 151089.118000,166420.105500 151143.220000,166298.254500 151211.571000,166127.184500 151259.095000,166026.410500 151305.462000,165928.074500 151349.337000,165848.246500 151386.821000,165783.121500 151430.102000,165713.152500 151510.142000,165592.644500 151544.728000,165544.121500 151627.696000,165440.598500 151681.994000,165378.879500 151706.415000,165352.168500 151749.806000,165306.683500 151789.688000,165268.785500 151818.478000,165245.558500 151869.056000,165208.980500 151908.876000,165183.488500 151948.665000,165166.137500 151996.158000,165150.504500 152012.669000,165146.101500 152013.467000,165007.192500 151976.837000,164697.027500 151976.400000,164666.965500 151975.415000,164599.730500 151975.634000,164578.152500 151975.962000,164546.105500 151976.087000,164534.183500 151975.868000,164523.387500 151975.837000,164504.058500 151975.728000,164454.918500 151975.087000,164420.902500 151974.587000,164388.355500 151974.150000,164360.387500 151973.900000,164344.090500 151974.283000,164310.228500 151977.634000,164287.637500 151983.618000,164209.965500 151985.525000,164193.855500 151987.040000,164189.871500 151990.040000,164185.090500 152001.056000,164165.480500 152026.696000,164123.371500 152059.025000,164073.808500 152088.665000,164025.152500 152130.478000,163954.746500 152166.321000,163894.777500 152214.618000,163813.996500 152227.884000,163793.371500 152243.540000,163769.027500 152297.275000,163678.668500 152303.056000,163668.965500 152355.743000,163580.902500 152385.478000,163530.683500 152394.587000,163516.871500 152402.915000,163504.433500 152404.696000,163502.074500 152508.409600,163342.544500 152657.933600,163114.813500 152665.712600,163103.048500 152770.720300,162940.244700 152780.919600,162925.034500 152869.139600,162789.204500 152875.103500,162780.879700 152872.029600,162774.834500 152634.475500,162308.892400 152629.259600,162295.104500 152621.699600,162207.737800 151967.320600,161904.469800 151965.084600,161903.433800 151005.015600,161468.249800 150997.066600,161464.656800 150989.835600,161495.608800 150984.044600,161527.288800 150980.679600,161553.368800 150979.298900,161572.654000 150978.520000,161599.283300 150978.159600,161692.713800 150978.039800,161792.017600 150976.951600,162017.283800 150971.471700,162029.483800 150969.481800,162105.048200 150783.210600,162347.537800 150688.824600,162484.743800 150661.369000,162503.404500 150610.632000,162526.603500 150575.507000,162553.018500 150532.036000,162593.352500 150494.213000,162644.148500 150459.040000,162719.145500 150446.190000,162780.133500 150441.837000,162837.517500 150450.138000,162904.717500 150467.757000,162960.516500 150409.106000,163141.437500 150380.748000,163452.645500 150393.113700,163658.347800 150390.641200,163674.579600 150402.681700,163680.353800 150515.551700,163738.873800 150527.471700,163743.693800 150645.151700,163804.363800 150646.321700,163808.773800 150512.733700,164072.195800 150502.614700,164093.358800 150449.656700,164211.614800 150442.171700,164237.303800 150363.914800,164720.007900 150361.041700,164738.923800 150179.537000,165654.136500 149926.133000,166009.311500 149835.002500,166163.775000 149729.157600,166633.164800 149956.199600,166887.263800 149914.708100,167054.132500 150211.036900,167133.510700 150229.557700,167175.923500 150213.682700,167239.331000 150615.850200,167345.164600 150656.225600,167309.907800 150696.524300,167062.392500 150738.025000,166844.402500 </gml:coordinates>
            </gml:LinearRing>
          </gml:outerBoundaryIs>
        </gml:Polygon>
        </TEST:msGeometry>
        <TEST:gid>1</TEST:gid>
        <TEST:zone_type>Zone 3</TEST:zone_type>
      </TEST:test_case>
    </gml:featureMember>
</wfs:FeatureCollection>

However a multipolygon intersects fails silently (this seems to be hinted at in #461 mentioning failing for multipoints...)

mapserv -nh QUERY_STRING="MAP=/ms4w/apps/ticket-6176-geos-wkt/mapfile.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=test_case&Filter=<Filter><Intersects><PropertyName>msGeometry</PropertyName><gml:MultiPolygon><gml:Polygon><gml:outerBoundaryIs><gml:LinearRing><gml:coordinates>151437.316900359 163762.595628729,151931.246516917 163567.321129159,151460.290370896 163096.364983138,151046.76790122 163521.374188084,151437.316900359 163762.595628729</gml:coordinates></gml:LinearRing></gml:outerBoundaryIs></gml:Polygon><gml:Polygon><gml:outerBoundaryIs><gml:LinearRing><gml:coordinates>150529.864814123 165991.022270877,151356.909753477 165715.340624425,151414.343429821 164784.915067652,151310.962812402 164199.091568943,150977.847489607 164199.091568943,150817.033195843 164819.375273459,151230.55566552 165106.543655179,150771.086254768 165301.818154748,150346.077049822 165497.092654318,150346.077049822 165497.092654318,150529.864814123 165991.022270877</gml:coordinates></gml:LinearRing></gml:outerBoundaryIs></gml:Polygon></gml:MultiPolygon></Intersects></Filter>"

@jmckenna
Copy link
Member

jmckenna commented Mar 12, 2021

Actually my first test above is a MULTIPOLYGON according to OGR:

OGRFeature(ttt):1
  ttt (String) = yup
  MULTIPOLYGON (((150529.864814123 165991.022270877,151356.909753477 165715.340624425,151414.343429821 164784.915067652,151310.962812402 164199.091568943,150977.847489607 164199.091568943,150817.033195843 164819.375273459,151230.55566552 165106.543655179,150771.086254768 165301.818154748,150346.077049822 165497.092654318,150346.077049822 165497.092654318,150529.864814123 165991.022270877)))

The second test I ran above was me trying to paste 2 multipolygon WKT into the same filter request, no wonder it fails ha.

Note: I could not get your filter coordinates to work, so I generated my own multipolygon coordinates , that lie inside your 'test_case' table's extents.

Anyway, not sure if my input helped, but i tried. Wishing all a nice weekend.

@jmckenna
Copy link
Member

ah my second filter was not a valid multipolgon...retrying...

@jmckenna
Copy link
Member

I keep failing with my Multipolygon tests here, darn. I have checked msautotest and there are no multipolygon WFS filter tests (multipoint tests do exist).

@jratike80
Copy link

What if you now wrap the simple polygon that works into a MultiPolygon with only one part? If it fails that might proof that MultiPolygons do not work at all in filters.

I guess that for Jeff the request to use would be

mapserv -nh QUERY_STRING="MAP=/ms4w/apps/ticket-6176-geos-wkt/mapfile.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=GetFeature&TYPENAME=test_case&FILTER=<Filter><Intersects><PropertyName>msGeometry</PropertyName><gml:MultiPolygon><gml:Polygon><gml:outerBoundaryIs><gml:LinearRing><gml:coordinates>150529.864814123 165991.022270877,151356.909753477 165715.340624425,151414.343429821 164784.915067652,151310.962812402 164199.091568943,150977.847489607 164199.091568943,150817.033195843 164819.375273459,151230.55566552 165106.543655179,150771.086254768 165301.818154748,150346.077049822 165497.092654318,150346.077049822 165497.092654318,150529.864814123 165991.022270877</gml:coordinates></gml:LinearRing></gml:outerBoundaryIs></gml:Polygon></gml:MultiPolygon></Intersects></Filter>"

@rouault rouault self-assigned this Jun 9, 2021
@rouault
Copy link
Contributor

rouault commented Jun 9, 2021

The GML is invalid. Each gml:Polygon should be wrapped inside a gml:polygonMember </gml:polygonMember>. The OGR GML parser is lax and ignores element that it doesn't know (gml:Polygon as direct child of gml:MultiPolygon), and thus interprets this as an empty multipolygon, which mapserver can't make much use of.

@rouault rouault closed this as completed Jun 9, 2021
rouault added a commit to rouault/mapserver that referenced this issue Jun 9, 2021
rouault added a commit that referenced this issue Jun 10, 2021
FLTGetSpatialComparisonCommonExpression(): avoid nullptr deref on invalid geometry (relates to #6176)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants