Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] [WFS provider] Add support for WFS 2.0 joins and other improvements #3018

Merged
merged 10 commits into from
May 13, 2016

Conversation

rouault
Copy link
Contributor

@rouault rouault commented Apr 21, 2016

Second part of qgis/QGIS-Enhancement-Proposals#53
(QEP 35: WFS provider enhancements)

  • URI parameter with sql with SELECT / FROM / JOIN / WHERE / ORDER BY clauses
  • handle WFS 2.0 joins
  • handle DateTime fields
  • enable "Only request features overlapping the view extent" by default (and memorize the settings)
  • rework DescribeFeatureType parsing to handle responses with several documents, and some support for attribute types being complexType
  • rework feature transfer between downloader and iterator so as to avoid uncontrolled RAM usage when the iterator cannot keep up with the downloader
  • turn on WAL journaling for better reader / writer concurrency
  • add retry logic based on the 'Max retry in case of tile request errors' setting (renamed 'Max retry in case of tile or feature request errors')
  • error to MessageBar in case of failed download
  • in progress dialog, add a "Hide" button to mask the dialog
  • improve automated testing
  • add testing of the GUI of QgsWFSSourceSelect

@mhugo
Copy link

mhugo commented Apr 21, 2016

Hi.
Good to see improvements in this field.

I am not very familiar with WFS 2.0. Is the SQL join syntax part of the standard or is it something that you added here ? It is a way for the client to use SQL rather than Filters ? To how much extent is it specific or compatible with other SQL dialects ?
I am thinking about virtual layers that use SQLite. I've been implementing a SQLite SQL parser (https://github.com/mhugo/qgis_vlayers/blob/master/qgssqlparser.yy) and finally decided it was not necessary for the task.
But if a new SQL parser is introduced here it would be good if we can reuse it in other parts of the code. For syntax analysis, and probably query transformations.
What do you think ?

@rouault
Copy link
Contributor Author

rouault commented Apr 21, 2016

Is the SQL join syntax part of the standard

WFS 2.0 has an optional feature where a server can declare its capabilities to do regular attribute joins or spatial joins. The choice of using SQL at the client side is because it is a standard way to do so, but the join is expressed in a WFS/FES specific way when sent to the server.

It is a way for the client to use SQL rather than Filters ?

Yes, much easier than typing XML Filter

To how much extent is it specific or compatible with other SQL dialects ?

I've implemented a subset of SQL that is sufficient to address WFS needs. But already with stuff that cannot be handled by WFS (like arbitrary expressions in SELECT fields or WHERE / ON conditions). And I've done it separately from the WFS provider, so it can hopefully be extended.

@nyalldawson
Copy link
Collaborator

@rouault

Just got to say - phenomenal work on level of unit tests included here!

@rouault
Copy link
Contributor Author

rouault commented Apr 21, 2016

Note: the Travis failure on Linux QT4 on PyQgsPalLabelingServer and on several tests on Mac are the same as the ones in master. A build on my branch https://travis-ci.org/rouault/Quantum-GIS/builds/124478952 based on a slightly older head pass.

@rouault rouault force-pushed the wfs_sql branch 2 times, most recently from f040a79 to 2013693 Compare April 26, 2016 21:19
@rouault
Copy link
Contributor Author

rouault commented Apr 26, 2016

Rebased to fix conflict with master

@rouault rouault force-pushed the wfs_sql branch 3 times, most recently from 2e5da2b to 0e692bc Compare May 2, 2016 10:46
@rouault
Copy link
Contributor Author

rouault commented May 2, 2016

Re-rebased on latest master + qgis.PyQt import fix. Would be nice to have this merged

@rouault rouault force-pushed the wfs_sql branch 2 times, most recently from b84936a to a6faf98 Compare May 3, 2016 13:11
@jef-n jef-n self-assigned this May 3, 2016
rouault added 10 commits May 12, 2016 23:23
- Fix one bug with Scintilla 2.8.X when focus out event does not
  disable blinking cursor
- Make QScintilla widget not capture Escape key so that it can
  propagate to parent.
…oin layers

And also fix handling srsDimension=3 on main GML geometry objects and not only posList
Useful for WFS provider whose setSubsetString() can potentially
modify fields if changing the SELECT fields
…ovements

Second part of qgis/QGIS-Enhancement-Proposals#53
(QEP 35: WFS provider enhancements)

- URI parameter with sql with SELECT / FROM / JOIN / WHERE / ORDER BY clauses
- handle WFS 2.0 joins
- handle DateTime fields
- enable "Only request features overlapping the view extent" by default (and memorize the settings)
- rework DescribeFeatureType parsing to handle responses with several documents, and some support for attribute types being complexType
- rework feature transfer between downloader and iterator so as to avoid uncontrolled RAM usage when the iterator cannot keep up with the downloader
- turn on WAL journaling for better reader / writer concurrency
- add retry logic based on the 'Max retry in case of tile request errors' setting (renamed 'Max retry in case of tile or feature request errors')
- error to MessageBar in case of failed download
- in progress dialog, add a "Hide" button to mask the dialog
- improve automated testing
- add testing of the GUI of QgsWFSSourceSelect
@jef-n
Copy link
Member

jef-n commented May 13, 2016

@rouault welcome :)

@rouault rouault merged commit d4d13a4 into qgis:master May 13, 2016
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants