Export selection from a large WFS layer fails #42049
Comments
LarsAtOmkeere
added
the
Bug
@LarsAtOmkeere that seems a very large layer, on a very slow service. How many features have you selected? Does it makes any difference if you load the layer with the native WFS client? @rduivenvoorde pinging you as author of "PDOK Services plugin" |
|
Hi Giovanni,
Thank you for your response. It is a large dataset (I think there approx. 8 million objects in there). I tried it multiple times and selected a few (< 10) or ~5500 features. The amount of features didn’t make a difference.
I hope this helps you in solving the bug.
Best regards,
Lars de Vries
Van: Giovanni Manghi <notifications@github.com>
Verzonden: vrijdag 5 maart 2021 09:57
Aan: qgis/QGIS <QGIS@noreply.github.com>
CC: Lars de Vries <larsdevries@omkeere.com>; Mention <mention@noreply.github.com>
Onderwerp: Re: [qgis/QGIS] export selection fails (#42049)
I added a layer 'pand' from the WFS layer BAG (https://geodata.nationaalgeoregister.nl/bag/wfs/v1_1) through the "PDOK Services plugin".
@LarsAtOmkeere<https://github.com/LarsAtOmkeere> that seems a very large layer, on a very slow service.
How many features have you selected?
Does it makes any difference if you load the layer with the native WFS client?
@rduivenvoorde<https://github.com/rduivenvoorde> pinging you as author of "PDOK Services plugin"
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#42049 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO2BOCXAKMPBTIEHYL44FA3TCCMHFANCNFSM4YUCP27Q>.
|
@LarsAtOmkeere does it makes any difference if you load that WFS layer with the QGIS native WFS client? |
|
So I tried it loading the data with the native client. This didn’t make a difference. It always shows the following dialog (dutch):
[cid:image001.png@01D711A9.5E438EF0]
This time I also tried writing to an existing Geopackage. When I do that, the process finishes. So the bug only seems to arise when writing to a new geopackage.
Van: Giovanni Manghi <notifications@github.com>
Verzonden: vrijdag 5 maart 2021 10:15
Aan: qgis/QGIS <QGIS@noreply.github.com>
CC: Lars de Vries <larsdevries@omkeere.com>; Mention <mention@noreply.github.com>
Onderwerp: Re: [qgis/QGIS] export selection fails (#42049)
The amount of features didn’t make a difference.
@LarsAtOmkeere<https://github.com/LarsAtOmkeere> does it makes any difference if you load that WFS layer with the QGIS native WFS client?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#42049 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO2BOCTQMBKS5CAGGEGERYTTCCOLBANCNFSM4YUCP27Q>.
|
@LarsAtOmkeere does not seems to make any difference for me on 3.18 on Linux. |
gioman
changed the title
gioman
added
WFS data provider
Vectors
|
And by the way, the export process is unkillable... |
|
For me it is possible to kill the process. When I click the right button (“Afbreken” in Dutch) it tells me the process is finished and the data is added to the map. When I click the left button (“Verbergen” in Dutch), the dialog disappears, but the process stays active, but does not finish. And I am no longer capable of getting the dialog back. From that moment on I cannot kill the process.
Van: Giovanni Manghi <notifications@github.com>
Verzonden: vrijdag 5 maart 2021 11:03
Aan: qgis/QGIS <QGIS@noreply.github.com>
CC: Lars de Vries <larsdevries@omkeere.com>; Mention <mention@noreply.github.com>
Onderwerp: Re: [qgis/QGIS] Export selection from a large WFS layer fails (#42049)
And by the way, the export process is unkillable...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#42049 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO2BOCQMJRY7QAWCEEFP5W3TCCT35ANCNFSM4YUCP27Q>.
|
one way or the other, is messy and not a good UX. |
|
@LarsAtOmkeere Well... it seems that leaving it doing its thing for an awful long amount of time it works as expected. I'm wondering if this is not just because the endpoint is very slow. |
|
I can’t judge that. Even a small dataset takes way to long. I don’t know whether the application is waiting on a message from the service. In my opinion that would be strange, since aborting the export tells me the task is finished.
Van: Giovanni Manghi <notifications@github.com>
Verzonden: vrijdag 5 maart 2021 13:22
Aan: qgis/QGIS <QGIS@noreply.github.com>
CC: Lars de Vries <larsdevries@omkeere.com>; Mention <mention@noreply.github.com>
Onderwerp: Re: [qgis/QGIS] Export selection from a large WFS layer fails (#42049)
@LarsAtOmkeere<https://github.com/LarsAtOmkeere> Well... it seems that leaving it doing its thing for an awful long amount of time it works as expected. I'm wondering if this is not just because the endpoint is very slow.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#42049 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AO2BOCXXT463BPERNZLIARDTCDEHXANCNFSM4YUCP27Q>.
|
|
@gioman @LarsAtOmkeere in my experience WFS is workable for smaller datasets or subsets. Note that in the capabilities you are instructed to only get 1000 buildings per time (which is actually pretty low for such a dataset... 10000 would be a better number), so doing larger area's is sometimes several tens of requests. In this case: a huge dataset with 8miljon records (all buildings in NL), it works if you select a small area, say 2000 buildings (panden). Also note that QGIS misses some parts sometimes, when requesting larger area's all grey buildings with a housenumber should be retrieved (pink features):
And even panning or zooming doe NOT give you the buildings anymore. QGIS has decided that in that area everything was received. I do not know WHAT QGIS triggers to stop requesting (as in: deciding it does not need to request anymore...: the number of sent records? @rouault can you maybe tell this?) Is QGIS 'guessing' an extent for the amount of features? So if the request extent is to large for the 'max step size' you will miss features? If you request the a smaller area (bantamstraat in Haarlem (there is a geocoder in the pdokservicesplugin)) QGIS will just sent you everything:
My guess is that if you keep the needed requests small (so start with a small area and pan around) it works better then trying to retrieve a whole city in one go. Selecting and then saving to file is an even trickier process, not sure why. I would think that the features are already on the clientside, but often you will see QGIS fire again a lot of requests... All in all the WFS for a dataset like the BAG is maybe not a good marriage in combi with QGIS Note that it is possible (from other sources) to retrieve the same dataset (for the whole of NL) as a postgis dump. We do have a lot of public services to test with. I think both Geoserver and Mapserver... |
|
@rduivenvoorde<https://github.com/rduivenvoorde> It was my intention to only download a dataset of approx.. 5000 features. I also tested with smaller datasets of just 20 features. But the bug still arose.
@Lars: I updated your comment here, apparently you replied via email and copied my WHOLE email in it...
|
|
With me here I can easily do a SMALL selection (in screenie for 230 buildings) and save to gpkg
But as soon as larger (? maybe 1000) you will see that QGIS is apparently looking up the features in the spatialite/sqlite database cache:
But at the same time it starts to sent requests to the server (as said: in steps of 1000) BUT WITHOUT BBOX: I really do not know WHY, maybe it tries to retrieve all features and then match against the ones in the 'selection'/cache? Which in this case is not very efficient (trying to retrieve 8.000.000 buildings, as we request withoug bbox....) Anyway: WFS is tricky business and QGIS is doing it's best, we have had a lot of different ways of doing this already, but there are always use cases which miss something. @LarsAtOmkeere as said: I think it is easier to download the postgis dump, load it in postgis and work with that... although off course you will never have the latest/greatest data version then.... |
|
One other test I did: I hardcoded the mShared->mPageSize = 10000 in the qgswfsprovider.cpp |
It seems to me that the problem is not certainly limited to the service/layer that is described in this ticket. I spent a very frustrating time testing heavy WFS layers (from a few different endpoints( on QGIS 3.18 and the experience has been very poor, both when it was time to just load a layer and also when I tried to select even a few features (even just 1) end then export them. I think we can't just say "big layers should not go in a WFS service", we should try to give at least an user experience that is at least usable, especially if the "competition" does a better job. Today I noticed:
It could have been just a very bad day... hopefully, but maybe not. |
…res by fids... and when they are already in the local cache. Fixes qgis#42049
…res by fids... and when they are already in the local cache. Fixes #42049
…res by fids... and when they are already in the local cache. Fixes qgis#42049
…res by fids... and when they are already in the local cache. Fixes #42049
…res by fids... and when they are already in the local cache. Fixes #42049
…res by fids... and when they are already in the local cache. Fixes #42049
…res by fids... and when they are already in the local cache. Fixes #42049
[3.16.4-Hannover, Windowss 10]
** BUG **
Progress dialog stays on screen and export process does not finish by itself.
How to Reproduce
QGIS and OS versions
QGIS-versie
3.16.4-Hannover
QGIS code revisie
654e76b
Gecompileerd tegen Qt
5.11.2
Gebruikt nu Qt
5.11.2
Gecompileerd tegen GDAL/OGR
3.1.4
Gebruikt nu GDAL/OGR
3.1.4
Gecompileerd tegen GEOS
3.8.1-CAPI-1.13.3
Gebruikt nu GEOS
3.8.1-CAPI-1.13.3
Gecompileerd tegen SQLite
3.29.0
Uitgevoerd met SQLite
3.29.0
PostgreSQL Client Versie
11.5
SpatiaLite-versie
4.3.0
QWT versie
6.1.3
QScintilla2 Versie
2.10.8
Gecompileerd tegen PROJ
6.3.2
Gebruikt nu PROJ
Rel. 6.3.2, May 1st, 2020
Versie OS
Windows 10 (10.0)
Actieve plug-ins van Python
b4udignl2;
pdokservicesplugin;
pdok_locatieserver_locator_filter;
SpreadsheetLayers;
xyToPoint;
db_manager;
MetaSearch;
processing
The text was updated successfully, but these errors were encountered: