Browser very slow when opening directory containing several GDB #53265

jfbourdon · 2023-05-29T16:16:11Z

What is the bug or the crash?

The browser takes a long time to show the content of a directory containing several ESRI File Geodatabase (up to about 100 seconds in my case for a folder with 132 GDB each containing about 20 layers and tables). During this time, QGIS is unresponsive and I can see via CPU and disk usage that it is doing "something". Once the directory opened, scrolling in the browser is slow and jaggy. Closing the directory restore the fluidity of the browser. Clicking on a GDB is also sluggish. Draging a layer is sluggish.

I looked at #51595 and #51848 but those issues don't correspond exactly to what I'm experiencing. The driver used (ESRI FileGDB or OpenFileGDB driver) doesn't seem to have too much of an impact in the loading time.

Based on all the QGIS versions I tested, I'm under the impression that the issue isn't with GDAL/OGR but rather with the new browser features introduced in 3.28 (see changelog).

Steps to reproduce the issue

In the Browser Panel, navigate to a folder containing tens of big GDB (doing so this with only a few GDB doesn't show as much)
Once opened, scroll in the browser to see that it isn't smooth
Open a GDB and drag-drop a layer onto the canvas to see the slugginess of the operation

The test can be performed with all the GDB inside all the zipped files available at https://diffusion.mern.gouv.qc.ca/Diffusion/RGQ/Vectoriel/Carte_Topo/Local/GRHQ/FGDB/. The times with this dataset aren't as bad as with the GDB I'm currently using, but the difference between near instant and 12 seconds stay pretty clear along with the jaggy scrolling .

Versions

I tested this with several version of QGIS:

QGIS	GDAL	Directory display time in browser
3.22.8	3.5.0	Near instant
3.22.16	3.6.2	Near instant
3.24.2	3.4.2	Near instant
3.26.0	3.5.0	Near instant
3.28.3	3.6.2	98 seconds
3.30.1	3.6.3	23 seconds
3.30.2	3.7.0	23 seconds

Supported QGIS version

I'm running a supported QGIS version according to the roadmap.

New profile

I tried with a new QGIS profile

Additional context

No response

…refs qgis#53265)

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes qgis#53265

nyalldawson · 2023-05-29T23:48:13Z

Thanks for the detailed bug report!

… drop values (refs qgis#53265)

…refs #53265)

… drop values (refs #53265)

…refs #53265)

… drop values (refs #53265)

…refs #53265)

… drop values (refs #53265)

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes #53265

jfbourdon · 2023-06-06T02:48:46Z

I downloaded the weekly snapshot (b9736bd) to test the fix and the improvement is very noticeable: smooth drag and drop, smooth scrolling and faster loading time for a folder full of GDB. However, for this last part, it's still several seconds slower than in <= 3.26. It essentialy only halved the times of 3.30 with 12 seconds for my original dataset and 6 seconds for the linked dataset.

Is there a way for me to try diagnose what is going on my machine? There must be some more requests made to each GDB compared to <= 3.26. Could it be a Windows specific issue?

jfbourdon · 2023-06-07T01:16:43Z

To be sure that it wasn't a weird issue with our corporate Windows environment at work, I tested the same QGIS versions and same datasets on my personal computer (also Windows 10) and I can confirm the same behaviour.

determine layer drop support This is rather tricky -- qt needs to know in advance whether a model item supports drops, so we need to determine whether we'll allow item drops as soon as we create any layer items in the browser. Unfortunately, determining this accurately requires opening a dataset for files associated with OGR vector layers. And this can be expensive to do, eg when a folder contains many GDB files. And since the item flags must be determined upfront, we **can't** do anything clever like delay determination of the drop capabilities until the user actually WANTS to drop content over a particular item. So, avoid the upfront cost of opening datasets by limiting ourselves to very cheap tests of files when populating directories. Specifically, we now only: - check if the file is read only - check if the associated GDAL driver reflects support for multiple layers in its metadata It's not perfect, but the best tradeoff we can do right now! Refs qgis#53265

nyalldawson · 2023-06-07T23:48:15Z

@jfbourdon

Should be fixed by #53398

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes #53265

determine layer drop support This is rather tricky -- qt needs to know in advance whether a model item supports drops, so we need to determine whether we'll allow item drops as soon as we create any layer items in the browser. Unfortunately, determining this accurately requires opening a dataset for files associated with OGR vector layers. And this can be expensive to do, eg when a folder contains many GDB files. And since the item flags must be determined upfront, we **can't** do anything clever like delay determination of the drop capabilities until the user actually WANTS to drop content over a particular item. So, avoid the upfront cost of opening datasets by limiting ourselves to very cheap tests of files when populating directories. Specifically, we now only: - check if the file is read only - check if the associated GDAL driver reflects support for multiple layers in its metadata It's not perfect, but the best tradeoff we can do right now! Refs qgis#53265

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes #53265

determine layer drop support This is rather tricky -- qt needs to know in advance whether a model item supports drops, so we need to determine whether we'll allow item drops as soon as we create any layer items in the browser. Unfortunately, determining this accurately requires opening a dataset for files associated with OGR vector layers. And this can be expensive to do, eg when a folder contains many GDB files. And since the item flags must be determined upfront, we **can't** do anything clever like delay determination of the drop capabilities until the user actually WANTS to drop content over a particular item. So, avoid the upfront cost of opening datasets by limiting ourselves to very cheap tests of files when populating directories. Specifically, we now only: - check if the file is read only - check if the associated GDAL driver reflects support for multiple layers in its metadata It's not perfect, but the best tradeoff we can do right now! Refs #53265

jfbourdon · 2023-06-12T19:34:40Z

Thanks @nyalldawson ! It now works as expected: fast and smooth!

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes #53265

jfbourdon added the Bug Either a bug report, or a bug fix. Let's hope for the latter! label May 29, 2023

agiudiceandrea added the QGIS Browser label May 29, 2023

nyalldawson self-assigned this May 29, 2023

rouault added a commit to rouault/QGIS that referenced this issue May 29, 2023

QgsBrowserGuiModel::flags(): cache value for faster browser redrawn (…

b2097f7

…refs qgis#53265)

nyalldawson mentioned this issue May 29, 2023

Cache database connection capabilities for browser items so we don't create a new connection on every call to the qt model flags #53268

Merged

rouault added a commit to rouault/QGIS that referenced this issue May 30, 2023

QgsBrowserGuiModel::flags(): take into number of providers for cached…

686de23

… drop values (refs qgis#53265)

rouault added a commit to rouault/QGIS that referenced this issue May 30, 2023

QgsBrowserGuiModel::flags(): take into number of providers for cached…

ce83b23

… drop values (refs qgis#53265)

nyalldawson mentioned this issue May 30, 2023

Browser very slow when a KML file comes into view #52075

Closed

2 tasks

nyalldawson pushed a commit that referenced this issue May 30, 2023

QgsBrowserGuiModel::flags(): cache value for faster browser redrawn (…

9d49d3a

…refs #53265)

nyalldawson pushed a commit that referenced this issue May 30, 2023

QgsBrowserGuiModel::flags(): take into number of providers for cached…

9023a67

… drop values (refs #53265)

qgis-bot pushed a commit that referenced this issue May 30, 2023

QgsBrowserGuiModel::flags(): cache value for faster browser redrawn (…

702333f

…refs #53265)

qgis-bot pushed a commit that referenced this issue May 30, 2023

QgsBrowserGuiModel::flags(): take into number of providers for cached…

cd4382e

… drop values (refs #53265)

nyalldawson pushed a commit that referenced this issue May 31, 2023

QgsBrowserGuiModel::flags(): cache value for faster browser redrawn (…

531d49f

…refs #53265)

nyalldawson pushed a commit that referenced this issue May 31, 2023

QgsBrowserGuiModel::flags(): take into number of providers for cached…

2796d27

… drop values (refs #53265)

nyalldawson closed this as completed in #53268 May 31, 2023

nyalldawson added a commit that referenced this issue May 31, 2023

Cache database connection capabilities for browser items so

3088bc5

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes #53265

qgis-bot pushed a commit that referenced this issue May 31, 2023

Cache database connection capabilities for browser items so

783152c

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes #53265

nyalldawson mentioned this issue Jun 7, 2023

Don't open datasets when populating browser directories to determine layer drop support #53398

Merged

nyalldawson added a commit that referenced this issue Jun 8, 2023

Cache database connection capabilities for browser items so

081eaa2

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes #53265

nyalldawson added a commit that referenced this issue Jun 9, 2023

Cache database connection capabilities for browser items so

fb8d8e6

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes #53265

nyalldawson added a commit that referenced this issue Jun 26, 2023

Cache database connection capabilities for browser items so

c7879b9

we don't create a new connection on every call to the qt model flags Greatly speeds up browser when a large number of files are visible Fixes #53265

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Browser very slow when opening directory containing several GDB #53265

Browser very slow when opening directory containing several GDB #53265

jfbourdon commented May 29, 2023 •

edited

nyalldawson commented May 29, 2023

jfbourdon commented Jun 6, 2023 •

edited

jfbourdon commented Jun 7, 2023 •

edited

nyalldawson commented Jun 7, 2023

jfbourdon commented Jun 12, 2023

Browser very slow when opening directory containing several GDB #53265

Browser very slow when opening directory containing several GDB #53265

Comments

jfbourdon commented May 29, 2023 • edited

What is the bug or the crash?

Steps to reproduce the issue

Versions

Supported QGIS version

New profile

Additional context

nyalldawson commented May 29, 2023

jfbourdon commented Jun 6, 2023 • edited

jfbourdon commented Jun 7, 2023 • edited

nyalldawson commented Jun 7, 2023

jfbourdon commented Jun 12, 2023

jfbourdon commented May 29, 2023 •

edited

jfbourdon commented Jun 6, 2023 •

edited

jfbourdon commented Jun 7, 2023 •

edited