Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setFilterExpression and QGIS Filter output different results #33454

Closed
Infiziert90 opened this issue Dec 18, 2019 · 6 comments
Closed

setFilterExpression and QGIS Filter output different results #33454

Infiziert90 opened this issue Dec 18, 2019 · 6 comments
Assignees
Labels
Bug

Comments

@Infiziert90
Copy link

@Infiziert90 Infiziert90 commented Dec 18, 2019

Describe the bug
setFilterExpression sorts out all NULL attributes, but the qgis internal filter function not.
So in pyqgis i get 8 rows and in qgis 4114 rows.

How to Reproduce
pyqgis:

layer = iface.activeLayer()
request = QgsFeatureRequest()
request = request.setFilterExpression('"COM"  NOT LIKE \'muster\'')
t = 0
for x in layer.getFeatures(request):
    t += 1
print(t)

Result: 8

Qgis Filter:
"COM" NOT LIKE 'muster'
Result: The where clause returned 4114 row(s).

QGIS and OS versions

QGIS version: 3.10.0-A Coruña
QGIS code revision: 6c816b4
Compiled against Qt: 5.11.2
Running against Qt: 5.11.2
Compiled against GDAL/OGR: 2.4.1
Running against GDAL/OGR: 2.4.1
Compiled against GEOS: 3.8.0-CAPI-1.13.1
Running against GEOS: 3.8.0-CAPI-1.13.1
Compiled against SQLite: 3.29.0
Running against SQLite: 3.29.0
PostgreSQL Client Version: 11.5
SpatiaLite Version: 4.3.0
QWT Version: 6.1.3
QScintilla2 Version: 2.10.8
Compiled against PROJ: 5.2.0
Running against PROJ: Rel. 5.2.0, September 15th, 2018
OS Version: Windows 10 (10.0)

@Infiziert90 Infiziert90 added the Bug label Dec 18, 2019
@roya0045

This comment has been minimized.

Copy link
Contributor

@roya0045 roya0045 commented Dec 19, 2019

That's strange because the set filter expression should convert that to a where clause in the provider iterator.

@Infiziert90

This comment has been minimized.

Copy link
Author

@Infiziert90 Infiziert90 commented Dec 19, 2019

Is the behavior with sorting out NULL intended?

@troopa81 troopa81 self-assigned this Jan 24, 2020
@troopa81

This comment has been minimized.

Copy link
Contributor

@troopa81 troopa81 commented Jan 24, 2020

@Infiziert90 I failed to reproduce with QGIS 3.10 and master. I have the same kind of filter with NULL values on a PostGres database and I get exactly the same number of feature with both ways.

Could you provide data and project so I can reproduce?

@troopa81 troopa81 added the Feedback label Jan 24, 2020
@Infiziert90

This comment has been minimized.

Copy link
Author

@Infiziert90 Infiziert90 commented Feb 10, 2020

Sorry for the late response ...

i tested it again with QGIS 3.10.2 and nothing changed.

My steps:

  1. Drag&Drop altlpar0.dbf into qgis
  2. open python console
  3. typed this in console ->
>>>layer = iface.activeLayer()
>>>request = QgsFeatureRequest()
>>>request = request.setFilterExpression('"COM"  NOT LIKE \'muster\'')
>>>t = 0
>>>for x in layer.getFeatures(request):
...    t += 1
>>>print(t)
0


>>>layer = iface.activeLayer()
>>>request = QgsFeatureRequest()
>>>request = request.setFilterExpression('"COM"  NOT LIKE \'bru\'')
>>>t = 0
>>>for x in layer.getFeatures(request):
...    t += 1
>>>print(t)
8
  1. right click altlpar0 and select the "Filter..." option
  2. type in "COM" NOT LIKE 'muster'
  3. Test: "The where clause returned 4132 row(s)."
  4. type in "COM" NOT LIKE 'bru'
  5. Test: "The where clause returned 4140 row(s)."

@troopa81
altlpar0.zip

@Infiziert90

This comment has been minimized.

Copy link
Author

@Infiziert90 Infiziert90 commented Feb 28, 2020

Same behavior with 3.12.0

@troopa81

This comment has been minimized.

Copy link
Contributor

@troopa81 troopa81 commented Mar 2, 2020

@Infiziert90 IMHO, it's not a QGIS issue (or maybe not an issue at all).

When you use setFilterExpression, the expression is evaluated by QGIS. When you use Filter... option, it delegates the filtering to the data provider, in your case OGR (gdal vector part), in my case the Postgres provider (that's why I failed to reproduce it the first place, we had different providers).

When QGIS evaluate expression like, it returns false (filter feature) if either left part or right part of the expression is null.

That's the same with Postgres

test_filterexpression=# select count(*) from altlpar0 limit1 where COM not like 'muster';
 count 
-------
     0
(1 row)

But that's different for OGR

$ ogrinfo -sql "select count(*) from altlpar0 where COM not like 'muster'" ~/work/tmp/bug_qgis_33454/altlpar0.dbf

INFO: Open of `/home/julien/work/tmp/bug_qgis_33454/altlpar0.dbf'
      using driver `ESRI Shapefile' successful.

Layer name: altlpar0
Geometry: None
Feature Count: 1
Layer SRS WKT:
(unknown)
COUNT_*: Integer (0.0)
OGRFeature(altlpar0):0
  COUNT_* (Integer) = 4132

My advice is to complete your filter to avoid ambiguity. For instance:

"COM" is not NULL and "COM" NOT LIKE 'muster

Or you can fill an issue in gdal arguing that the subset filter behavior differs with the PostGres and QGIS one.

@troopa81 troopa81 closed this Mar 2, 2020
@troopa81 troopa81 removed the Feedback label Mar 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.