WFS request with chinese keywords does not have any result #5215

Merged
merged 1 commit into from Jan 6, 2016

Conversation

Projects
None yet
2 participants
@tbonfort
Member

tbonfort commented Jan 5, 2016

My WFS request is the following code:

          <?xml version='1.0' encoding='UTF-8'?>
            <wfs:GetFeature  service='WFS' version='1.0.0' 
            xmlns:wfs='http://www.opengis.net/wfs' 
            xmlns:gml='http://www.opengis.net/gml' 
            xmlns:ogc='http://www.opengis.net/ogc' 
           xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' 
            xsi:schemaLocation='http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.0.0/wfs.xsd'>           
 <wfs:Query typeName='district' srsName='EPSG:3830'>
            <ogc:Filter>
            <ogc:PropertyIsEqualTo>
            <ogc:PropertyName>TField</ogc:PropertyName>
            <ogc:Literal>高新区</ogc:Literal>
            </ogc:PropertyIsEqualTo>
           </ogc:Filter>
           </wfs:Query>
           </wfs:GetFeature>

I want to search the feature that the value of field named TField is 高新区. I set the the layer's encoding to GBK, ant the above request returns the message that No matching record(s) found. however, if I set the keyword to english words or number, then I can get the result. The code above works well on Geoserver WFS, but can't got result from mapserver' WFS. Additional, "高新区" is Chinese words.

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Jan 4, 2016

Member

Please give more info:

  • What is your mapserver version
  • What is the encoding/type of your underlaying datasource
  • How are you setting the layer encoding in the mapfile
Member

tbonfort commented Jan 4, 2016

Please give more info:

  • What is your mapserver version
  • What is the encoding/type of your underlaying datasource
  • How are you setting the layer encoding in the mapfile
@bruce6135

This comment has been minimized.

Show comment
Hide comment
@bruce6135

bruce6135 Jan 4, 2016

Thanks for your atention. My mapserver version is 7.0.0, which is build in ms4w 3.1.0. My datasource is ESRI Shapefile format, and I modified the data source's encode to GBK. In my mapfile, I set the Map and Layer's encoding to GBK.

Thanks for your atention. My mapserver version is 7.0.0, which is build in ms4w 3.1.0. My datasource is ESRI Shapefile format, and I modified the data source's encode to GBK. In my mapfile, I set the Map and Layer's encoding to GBK.

@bruce6135

This comment has been minimized.

Show comment
Hide comment
@bruce6135

bruce6135 Jan 5, 2016

In order to help you understand my question, I copied the log message from mapserver's log file.
Here is a wfs request that got the result features:

[Tue Jan 05 10:52:43 2016].848000 CGI Request 1 on process 3464
[Tue Jan 05 10:52:43 2016].848000 msWFSParseRequest(): WFS post request: <wfs:GetFeature service='WFS' version='1.0.0' xmlns:wfs='http://www.opengis.net/wfs' xmlns:gml='http://www.opengis.net/gml' xmlns:ogc='http://www.opengis.net/ogc' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xsi:schemaLocation='http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.0.0/wfs.xsd'><wfs:Query typeName='district' srsName='EPSG:3830'>ogc:Filterogc:PropertyIsEqualToogc:PropertyNameTField/ogc:PropertyNameogc:Literal440402/ogc:Literal/ogc:PropertyIsEqualTo/ogc:Filter/wfs:Query/wfs:GetFeature
[Tue Jan 05 10:52:43 2016].851000 FLTLayerApplyPlainFilterToLayer(): ([TField] =440402)
[Tue Jan 05 10:52:43 2016].865000 mapserv request processing time (msLoadMap not incl.): 0.017s
[Tue Jan 05 10:52:43 2016].865000 msFreeMap(): freeing map at 05C1A5B0.

And here is a wfs request that can not got the result features:

[Tue Jan 05 10:19:25 2016].525000 CGI Request 1 on process 3220
[Tue Jan 05 10:19:25 2016].525000 msWFSParseRequest(): WFS post request: <wfs:GetFeature service='WFS' version='1.0.0' xmlns:wfs='http://www.opengis.net/wfs' xmlns:gml='http://www.opengis.net/gml' xmlns:ogc='http://www.opengis.net/ogc' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xsi:schemaLocation='http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.0.0/wfs.xsd'><wfs:Query typeName='district' srsName='EPSG:3830'>ogc:Filterogc:PropertyIsEqualToogc:PropertyNameTField/ogc:PropertyNameogc:Literal高新区/ogc:Literal/ogc:PropertyIsEqualTo/ogc:Filter/wfs:Query/wfs:GetFeature
[Tue Jan 05 10:19:25 2016].526000 FLTLayerApplyPlainFilterToLayer(): ("[TField]" ="高新区")
[Tue Jan 05 10:19:25 2016].527000 msQueryByFilter(): Search returned no results. No matching record(s) found.
[Tue Jan 05 10:19:25 2016].527000 mapserv request processing time (msLoadMap not incl.): 0.002s
[Tue Jan 05 10:19:25 2016].527000 msFreeMap(): freeing map at 05B1A5B0.

I don't know how to resolve this problem, can anybody tell me how to do?

In order to help you understand my question, I copied the log message from mapserver's log file.
Here is a wfs request that got the result features:

[Tue Jan 05 10:52:43 2016].848000 CGI Request 1 on process 3464
[Tue Jan 05 10:52:43 2016].848000 msWFSParseRequest(): WFS post request: <wfs:GetFeature service='WFS' version='1.0.0' xmlns:wfs='http://www.opengis.net/wfs' xmlns:gml='http://www.opengis.net/gml' xmlns:ogc='http://www.opengis.net/ogc' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xsi:schemaLocation='http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.0.0/wfs.xsd'><wfs:Query typeName='district' srsName='EPSG:3830'>ogc:Filterogc:PropertyIsEqualToogc:PropertyNameTField/ogc:PropertyNameogc:Literal440402/ogc:Literal/ogc:PropertyIsEqualTo/ogc:Filter/wfs:Query/wfs:GetFeature
[Tue Jan 05 10:52:43 2016].851000 FLTLayerApplyPlainFilterToLayer(): ([TField] =440402)
[Tue Jan 05 10:52:43 2016].865000 mapserv request processing time (msLoadMap not incl.): 0.017s
[Tue Jan 05 10:52:43 2016].865000 msFreeMap(): freeing map at 05C1A5B0.

And here is a wfs request that can not got the result features:

[Tue Jan 05 10:19:25 2016].525000 CGI Request 1 on process 3220
[Tue Jan 05 10:19:25 2016].525000 msWFSParseRequest(): WFS post request: <wfs:GetFeature service='WFS' version='1.0.0' xmlns:wfs='http://www.opengis.net/wfs' xmlns:gml='http://www.opengis.net/gml' xmlns:ogc='http://www.opengis.net/ogc' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xsi:schemaLocation='http://www.opengis.net/wfs http://schemas.opengis.net/wfs/1.0.0/wfs.xsd'><wfs:Query typeName='district' srsName='EPSG:3830'>ogc:Filterogc:PropertyIsEqualToogc:PropertyNameTField/ogc:PropertyNameogc:Literal高新区/ogc:Literal/ogc:PropertyIsEqualTo/ogc:Filter/wfs:Query/wfs:GetFeature
[Tue Jan 05 10:19:25 2016].526000 FLTLayerApplyPlainFilterToLayer(): ("[TField]" ="高新区")
[Tue Jan 05 10:19:25 2016].527000 msQueryByFilter(): Search returned no results. No matching record(s) found.
[Tue Jan 05 10:19:25 2016].527000 mapserv request processing time (msLoadMap not incl.): 0.002s
[Tue Jan 05 10:19:25 2016].527000 msFreeMap(): freeing map at 05B1A5B0.

I don't know how to resolve this problem, can anybody tell me how to do?

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Jan 5, 2016

Member

Can you try the attached patch? It would be nice if you could provide a sample shapefile so we can include this check in our test suite

Member

tbonfort commented Jan 5, 2016

Can you try the attached patch? It would be nice if you could provide a sample shapefile so we can include this check in our test suite

@bruce6135

This comment has been minimized.

Show comment
Hide comment
@bruce6135

bruce6135 Jan 5, 2016

I can not attach a patch in the comment, so I upload the file into oneDrive.
The link is :
https://onedrive.live.com/redir?resid=660C89D3266533C6!109&authkey=!AIAeACYiK62s0rY&ithint=file%2czip
https://onedrive.live.com/redir?resid=660C89D3266533C6!110&authkey=!AOoFQdF6ItRjpv4&ithint=file%2cmap
My data's EPSG is 4326.
When I enter the URL like this:

http://localhost:88/cgi-bin/mapserv.exe?map=D:/ms4w/Apache/htdocs/map/query.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=getFeature&TYPENAME=city&Filter=<Filter><PropertyIsEqualTo><PropertyName>NAME</PropertyName><Literal>塔河县</Literal></PropertyIsEqualTo></Filter>

then show the following:

<gml:boundedBy>
<gml:null>missing</gml:null>
</gml:boundedBy>

The information in the log file is :
[Tue Jan 05 23:07:33 2016].399000 CGI Request 1 on process 4908
[Tue Jan 05 23:07:33 2016].400000 FLTLayerApplyPlainFilterToLayer(): ("[NAME]" ="塔河县")
[Tue Jan 05 23:07:33 2016].438000 msQueryByFilter(): Search returned no results. No matching record(s) found.
[Tue Jan 05 23:07:33 2016].438000 mapserv request processing time (msLoadMap not incl.): 0.039s
[Tue Jan 05 23:07:33 2016].438000 msFreeMap(): freeing map at 05BAA990.

I can not attach a patch in the comment, so I upload the file into oneDrive.
The link is :
https://onedrive.live.com/redir?resid=660C89D3266533C6!109&authkey=!AIAeACYiK62s0rY&ithint=file%2czip
https://onedrive.live.com/redir?resid=660C89D3266533C6!110&authkey=!AOoFQdF6ItRjpv4&ithint=file%2cmap
My data's EPSG is 4326.
When I enter the URL like this:

http://localhost:88/cgi-bin/mapserv.exe?map=D:/ms4w/Apache/htdocs/map/query.map&SERVICE=WFS&VERSION=1.0.0&REQUEST=getFeature&TYPENAME=city&Filter=<Filter><PropertyIsEqualTo><PropertyName>NAME</PropertyName><Literal>塔河县</Literal></PropertyIsEqualTo></Filter>

then show the following:

<gml:boundedBy>
<gml:null>missing</gml:null>
</gml:boundedBy>

The information in the log file is :
[Tue Jan 05 23:07:33 2016].399000 CGI Request 1 on process 4908
[Tue Jan 05 23:07:33 2016].400000 FLTLayerApplyPlainFilterToLayer(): ("[NAME]" ="塔河县")
[Tue Jan 05 23:07:33 2016].438000 msQueryByFilter(): Search returned no results. No matching record(s) found.
[Tue Jan 05 23:07:33 2016].438000 mapserv request processing time (msLoadMap not incl.): 0.039s
[Tue Jan 05 23:07:33 2016].438000 msFreeMap(): freeing map at 05BAA990.

tbonfort added a commit to mapserver/msautotest_DEPRECATED that referenced this pull request Jan 6, 2016

@tbonfort tbonfort merged commit 100e6af into mapserver:branch-7-0 Jan 6, 2016

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Jan 6, 2016

Member

Pushed the fix to branch-7-0, along with an autotest in mapserver/msautotest_DEPRECATED@b52c73c . Without the current fix, the autotest result is

<gml:boundedBy>
<gml:null>missing</gml:null>
</gml:boundedBy>

instead of the correct feature.

Thanks for the testcase @bruce6135 , you can delete the data now, I was able to create the autotest with an exisiting dataset

Member

tbonfort commented Jan 6, 2016

Pushed the fix to branch-7-0, along with an autotest in mapserver/msautotest_DEPRECATED@b52c73c . Without the current fix, the autotest result is

<gml:boundedBy>
<gml:null>missing</gml:null>
</gml:boundedBy>

instead of the correct feature.

Thanks for the testcase @bruce6135 , you can delete the data now, I was able to create the autotest with an exisiting dataset

@bruce6135

This comment has been minimized.

Show comment
Hide comment
@bruce6135

bruce6135 Jan 6, 2016

Congratulations for you fixed this problem, and thanks for you help. I will try to compile in my computer to fix this problem.

Congratulations for you fixed this problem, and thanks for you help. I will try to compile in my computer to fix this problem.

@bruce6135

This comment has been minimized.

Show comment
Hide comment
@bruce6135

bruce6135 Jan 8, 2016

Sombody helped me to find another method. In QGIS, saving the layer as another layer and set the new layer's code to UTF-8.The new layer's property sheet is now in UTF-8 code and shows correctly. Then I can query chinese words with WFS service. The disadvantage of this method is that UTF-8 is not contain all the chinese words, by save as the layer, some chinese words will be garbled and must be reenter with the correct content. However, those garbled words usually is not often used and account for a small proportion of chinese words.

Sombody helped me to find another method. In QGIS, saving the layer as another layer and set the new layer's code to UTF-8.The new layer's property sheet is now in UTF-8 code and shows correctly. Then I can query chinese words with WFS service. The disadvantage of this method is that UTF-8 is not contain all the chinese words, by save as the layer, some chinese words will be garbled and must be reenter with the correct content. However, those garbled words usually is not often used and account for a small proportion of chinese words.

@tbonfort

This comment has been minimized.

Show comment
Hide comment
@tbonfort

tbonfort Jan 8, 2016

Member

I doubt this has something to do with UTF-8 not supporting those glyphs. You are probably giving qgis an incorrect version of gbk (with respect to the version actually used by your dataset) when asking it to convert it to utf8.
ps: always use utf-8 and you'll never have to deal with encoding issues again :)

Member

tbonfort commented Jan 8, 2016

I doubt this has something to do with UTF-8 not supporting those glyphs. You are probably giving qgis an incorrect version of gbk (with respect to the version actually used by your dataset) when asking it to convert it to utf8.
ps: always use utf-8 and you'll never have to deal with encoding issues again :)

@tbonfort tbonfort deleted the tbonfort:issues/5215-encoding-before-filter branch Jun 3, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment