[OGR provider] Allow opening (GeoPackage) datasets with many layers #5409
Conversation
rouault
force-pushed
the
fix_gpkg_many_layers
branch
from
f6cee61
to
dc41794
Compare
|
Contact me if you need a huge gpkg to test |
|
Hi, Thanks for this work
A stacktrace (I think more related to the multithread Data Source Manager...) but crashing QGIS: `/home/richard/bin/qgis_all/pr/debug/lib/qgis/plugins/libogrprovider.so(+0x9649c)[0x7fe84662f49c] ` |
I couldn't reproduce a crash, but the slowness yes. Which I've fixed with 129a9e2 . I should note also that best performance is reached with GDAL trunk as I've pushed a bunch of optimizations for faster opening of GeoPackage with a large number of layers, and "lazy" opening of layers. For example a test GeoPackage with 1000 layers opens in 5 seconds with GDAL 2.2, and 0.5s with GDAL trunk. On your GeoPackage with 144 layers, GDAL 2.2 in 300 ms, GDAL trunk in 160 ms.
I couldn't reproduce this issue. Speed is the same for me.
You can ignore this. This is probably due to the browser trying to open at some point the .gpkg as a raster one. |
|
Hi,
I compiled gdal master and QGIS master and can confirm that larger
geopackage now open much faster. We have a geopackage with cadastral
data (56 layers, several thousand features each). Before it took several
seconds until the dialogue with the layer list opened. Now this layer
list opens instantly. Also from USB stick it opens instantly. Haven't
tested network drive yet.
Good work - thanks a lot, Even!
Andreas
On 2017-10-20 13:37, Even Rouault wrote:
@rduivenvoorde [1]
> if I try to load the giant gpkg via the Data Source Manager, I can load 1 layer but with more it either crashes QGIS (stracktrace below), or just does not end (without any message, running 100% on 1 cpu core)
I couldn't reproduce a crash, but the slowness yes. Which I've fixed with 129a9e2 [2] .
I should note also that best performance is reached with GDAL trunk as I've pushed a bunch of optimizations for faster opening of GeoPackage with a large number of layers, and "lazy" opening of layers. For example a test GeoPackage with 1000 layers opens in 5 seconds with GDAL 2.2, and 0.5s with GDAL trunk. On your GeoPackage with 144 layers, GDAL 2.2 in 300 ms, GDAL trunk in 160 ms.
> if I drag/drop de gpkg in the layermanager, the FIRST time it is just fast, and QGIS shows me the 'select a layer' dialog, and I can even select all 143 layers.
> BUT a second or third time, this same action takes a very long time??
I couldn't reproduce this issue. Speed is the same for me.
> another observation: when I browse to the gpkg I see
> ERROR 4: `/home/richard/z/17/rivm/20170926_grootmodel/oudemodel/TotalGammaDoseRate.gpkg' not recognized as a supported file format.
You can ignore this. This is probably due to the browser trying to open at some point the .gpkg as a raster one.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub [3], or mute the thread [4].
|
Currently each time you instanciate a QgsOgrProvider layer, a GDAL dataset is created. In the case of GeoPackage, this means a SQLite connection and a file handle. As GDAL enables Spatialite function on GeoPackage connections, we are bound to Spatialite limits, and Spatialite has a hard limit on a maximum of 64 simultaneous connections. Thus we cannot open more than 64 layers of the same GeoPackage. This commits enables sharing of the same GDALDataset object among several QgsOgrProvider object. Care is made to reuse a GDALDataset object only if the QgsOgrProvider do not point to the same layer. Mutexes are also taken to allow safe instanciation and use of QgsOgrProvider objects from multiple threads (but a same QgsOgrProvider should not be used by more than one thread at a time)
…om the data source manager
rouault
force-pushed
the
fix_gpkg_many_layers
branch
from
129a9e2
to
c9b0a2b
Compare
Currently each time you instanciate a QgsOgrProvider layer, a GDAL dataset is
created. In the case of GeoPackage, this means a SQLite connection and a file
handle. As GDAL enables Spatialite function on GeoPackage connections, we are
bound to Spatialite limits, and Spatialite has a hard limit on a maximum of
64 simultaneous connections. Thus we cannot open more than 64 layers of the
same GeoPackage.
This commits enables sharing of the same GDALDataset object among several
QgsOgrProvider object. Care is made to reuse a GDALDataset object only if the
QgsOgrProvider do not point to the same layer. Mutexes are also taken to
allow safe instanciation and use of QgsOgrProvider objects from multiple
threads (but a same QgsOgrProvider should not be used by more than one thread
at a time)
Checklist
fixes #11111in the commit message next to the description[FEATURE]in the commit message[needs-docs]in the commit message and containt sufficient information in the commit message to be documentedscripts/prepare-commit.shscript before each commit