Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve handling of missing spatial indexes in Processing #30530

Closed
kannes opened this issue Jul 3, 2019 · 3 comments · Fixed by #36570
Closed

Improve handling of missing spatial indexes in Processing #30530

kannes opened this issue Jul 3, 2019 · 3 comments · Fixed by #36570
Assignees

Comments

@kannes
Copy link
Contributor

kannes commented Jul 3, 2019

Feature description.
Many GIS algorithms heavily benefit from spatial indexes. If there is no spatial index, any operations that rely on spatial relationships between geometries will be unnecessarily slow.

For example a Count points in polygon comparison:

  • 200,000 points in a Shapefile without spatial index: 30 seconds
  • Same Shapefile but now with a .qix spatial index: 3 seconds

Processing should warn, like it nowadays does for CRS mismatches, if there is no spatial index for operations that would benefit from it.

Alternatively Processing could generate indexes internally but I don't know anything about how QGIS handles this for its internal layer structure. It would be a waste if an index was created every time an algorithm ran, maybe they could be stored in memory.

This could greatly benefit many users and lead to QGIS being faster in random comparisons magically.

Additional context.
I just realised, after years of QGIS usage, that QGIS does not write a spatial index when exporting shapefiles (see #30528). This means I wasted countless hours over the years waiting for algorithms to finish just because there was no spatial index being used. :o)

@saberraz
Copy link
Contributor

saberraz commented Jul 3, 2019

One of the reason, spatial indexing is optional and somewhat hidden under layer properties, is due to the file/folder permission. Usually the index file sits with the file/geodatabase. If user has limited access/permission to the folder/schema, the index creation will throw up lots of errors.

@kannes
Copy link
Contributor Author

kannes commented Jul 3, 2019

Great point, I did not consider that. I am sure that many typical work environments for QGIS rely on networked, read-only data.

Highlighting the absence of a spatial index would still be useful then, the users would have to talk to their admins for (and hey, this would also reduce network traffic :) ).

@kannes
Copy link
Contributor Author

kannes commented Jul 3, 2019

@rouault just told me that there is a layer capability attribute OLCFastSpatialFilter exposed in the GDAL API that could be used: https://gdal.org/api/ogrlayer_cpp.html#_CPPv4N8OGRLayer14TestCapabilityEPKc

nyalldawson added a commit to nyalldawson/QGIS that referenced this issue May 20, 2020
Provides warnings in Processing algorithms when running on layers
which don't have a spatial index present (e.g. geojson files) and
where performance will be severely hurt as a result

Fixes qgis#30530
@nyalldawson nyalldawson self-assigned this May 20, 2020
nyalldawson added a commit that referenced this issue May 20, 2020
Provides warnings in Processing algorithms when running on layers
which don't have a spatial index present (e.g. geojson files) and
where performance will be severely hurt as a result

Fixes #30530
nyalldawson added a commit to nyalldawson/QGIS that referenced this issue May 26, 2020
Provides warnings in Processing algorithms when running on layers
which don't have a spatial index present (e.g. geojson files) and
where performance will be severely hurt as a result

Fixes qgis#30530

(cherry picked from commit f20801a)
nyalldawson added a commit that referenced this issue Jun 19, 2020
Provides warnings in Processing algorithms when running on layers
which don't have a spatial index present (e.g. geojson files) and
where performance will be severely hurt as a result

Fixes #30530

(cherry picked from commit f20801a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants