Very long loading times when opening the same FileGDB the second time

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)
Original Redmine Issue: 19968
Affected QGIS version: 3.3(master)
Redmine category:data_provider/ogr

---

The problem and how to reproduce:

1. I have a FileGDB which I drag into QGIS (14 seconds).
2. I select all layers and press OK (7 seconds).
3. I remove all the layers (empty layer list and this process is slow too).
4. I drag the same FileGDB into QGIS again (14 seconds).
5. I select all layers and press OK (I was disturbed by a colleague after about 2.5 minutes and all layers was loaded when I came back after 15 minutes).

As seen on the above timings loading the same FileGDB the second time is magnitudes slower than the first time around.

All I can see is that a lot of locking/unlocking of the files @GDB_Items@ and @GDB_ItemTypes@ during all the processes.

It might be related to issue #26232 where Even Rouault did some optimization.

---

I have attached the test data set used for the timings above (the same as used in issue #26232).
My tests was done on a Laptop with an SSD hard-drive but the issue is also present on my workstation.
I have installed QGIS nightly using OSGeo4W x64 and the package @gdal-filegdb@ is also installed.
I have ArcMap 10.6 and ArcGIS Pro 2.2 installed.

---

• testdata_2.gdb.zip (Casper Børgesen)

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Casper Børgesen wrote:

I have a FileGDB which I drag into QGIS (14 seconds).

I select all layers and press OK (7 seconds).

I remove all the layers (empty layer list and this process is slow too).

I drag the same FileGDB into QGIS again (14 seconds).

I select all layers and press OK (I was disturbed by a colleague after about 2.5 minutes and all layers was loaded when I came back after 15 minutes).

I wanted to see if ESRI had the same problem and I repeated the test in ArcGIS Pro:

• Step 1 and 2 is done in about 14 seconds.
• Step 3 is done almost instantly.
• Step 4 and 5 is also done in about 14 seconds.

QGIS seems to keep a lock on all feature types when the FileGDB is open, where ArcGIS Pro doesn't. I think that could be the cause to why QGIS is slower when removing all the layers.

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

Does it work differently in QGIS 2.18?

---

• status_id was changed from Open to Feedback

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Giovanni Manghi wrote:

Does it work differently in QGIS 2.18?

I have just tried on the two machines and QGIS 2.18 has the issue already in step 2.

Would it help if I recorded a video of what I am doing along with all the file activity in the @gdb@ folder?

---

• status_id was changed from Feedback to Open

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

Casper Børgesen wrote:

Giovanni Manghi wrote:

Does it work differently in QGIS 2.18?

I have just tried on the two machines and QGIS 2.18 has the issue already in step 2.

so not a regression.

Would it help if I recorded a video of what I am doing along with all the file activity in the @gdb@ folder?

yes please.

---

• priority_id was changed from High to Normal
• regression was changed from 1 to 0

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Giovanni Manghi wrote:

yes please.

I have uploaded a short video to youtube where you can see whats happening in QGIS and in the FileGDB folder when loading and removing the FileGDB from QGIS.

https://www.youtube.com/watch?v=hvcPVObfBFA

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

Just tested on master/linux and the operations as you show them in the screencast are basically instantaneous.

---

• status_id was changed from Open to Feedback

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Giovanni Manghi wrote:

Just tested on master/linux and the operations as you show them in the screencast are basically instantaneous.

I am only on Windows so I can't provide feedback regarding Linux.

---

• status_id was changed from Feedback to Open

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Giovanni Manghi wrote:

Just tested on master/linux and the operations as you show them in the screencast are basically instantaneous.

I just uninstalled the OGR FileGDB driver and reran the tests. Without the OGR FileGDB driver I probably get the same instantaneous loading like you do.

I guess that you don't have the OGR FileGDB driver installed (since I expect it to be Windows only?).

I have tried the same tests with and without the OGR FileGDB driver installed on my private PC (which isn't infected by any ESRI products) and I get the same results, with the driver its very slow and without its very fast. In most cases I need the write capability of the OGR FileGDB driver and I haven't found an easy way to select which driver to use - besides (un)installing :s

So I think this issue is related to the OGR FileGDB driver (or how it is used by QGIS).

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

Casper Børgesen wrote:

Giovanni Manghi wrote:

Just tested on master/linux and the operations as you show them in the screencast are basically instantaneous.

I just uninstalled the OGR FileGDB driver and reran the tests. Without the OGR FileGDB driver I probably get the same instantaneous loading like you do.

I guess that you don't have the OGR FileGDB driver installed (since I expect it to be Windows only?).

I have tried the same tests with and without the OGR FileGDB driver installed on my private PC (which isn't infected by any ESRI products) and I get the same results, with the driver its very slow and without its very fast. In most cases I need the write capability of the OGR FileGDB driver and I haven't found an easy way to select which driver to use - besides (un)installing :s

So I think this issue is related to the OGR FileGDB driver (or how it is used by QGIS).

I just tested with both the proprietary "ogr filegdb" (esri fliegdb) available to be installed manually with osgeo4w and the open source one (openFileGDB) is installed by default (as is part of ogr) on any operating system when installing QGIS. On Windows 10/QGIS 3.2 (osgeo4w) I can't notice any delay or long wait when adding/removing/re-adding your datasource.

Have you tried on a clean profile?

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Giovanni Manghi wrote:

Have you tried on a clean profile?

What do you mean by a clean profile?

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

Casper Børgesen wrote:

Giovanni Manghi wrote:

Have you tried on a clean profile?

What do you mean by a clean profile?

In QGSI3 you can have multiple profiles (settings > user profiles). Testing in a new profile is necessary to exclude issues caused by 3rd party plugins and/or legacy configurations.

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Giovanni Manghi wrote:

Casper Børgesen wrote:

Giovanni Manghi wrote:

Have you tried on a clean profile?

What do you mean by a clean profile?

In QGSI3 you can have multiple profiles (settings > user profiles). Testing in a new profile is necessary to exclude issues caused by 3rd party plugins and/or legacy configurations.

I have created a new profile and using that profile I still get the same slow results :(

[qgib]

Author Name: Andrea Giudiceandrea (@agiudiceandrea)

---

I can reproduce the issue on Windows 7 64 bit (core i5, 8GB) with QGIS 3.3.0-Master (3032af2) (qgis-dev-3.3.0-84) 64 bit.

The issue only happens when the testdata_2.gdb directory (extracted from the provided zip file) is dragged and dropped in the map or when the layers are added through "Add vector layer | Data Source Manager | Vector" / "Source Type: Directory" and only if the GDAL/OGR FileGDB driver plugin (gdal-filegdb 2.2.4-2) is installed (regardless of the Source format chosen among "OpenFileGDB" and "ESRI FileGDB").

I can also confirm that the slowdowns seem related to the creation of lock files in the testdata_2.gdb directory during the import/removal operations.

It doesn't happen if the GDAL/OGR FileGDB driver plugin is not installed.

It doesn't happen if the provided testdata_2.gdb.zip file (not the extracted testdata_2.gdb directory) is dragged and dropped into the map (regardless the GDAL/OGR FileGDB driver plugin is installed or not).

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

Andrea Giudiceandrea wrote:

I can reproduce the issue on Windows 7 64 bit (core i5, 8GB) with QGIS 3.3.0-Master (3032af2) (qgis-dev-3.3.0-84) 64 bit.

The issue only happens when the testdata_2.gdb directory (extracted from the provided zip file) is dragged and dropped in the map or when the layers are added through "Add vector layer | Data Source Manager | Vector" / "Source Type: Directory" and only if the GDAL/OGR FileGDB driver plugin (gdal-filegdb 2.2.4-2) is installed (regardless of the Source format chosen among "OpenFileGDB" and "ESRI FileGDB").

weird, I'm pretty sure I tested this scenario on a clean environment (VM snapshot) and I could not confirm. Anyway, if this observation is right then the title and description of this ticket should be changed to better reflect the source of the issue.

---

• status_id was changed from Open to Feedback

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Giovanni Manghi wrote:

... Anyway, if this observation is right then the title and description of this ticket should be changed to better reflect the source of the issue.

I am not allowed to change anything related to the question or subject.

---

• status_id was changed from Feedback to Open

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

Casper Børgesen wrote:

Giovanni Manghi wrote:

... Anyway, if this observation is right then the title and description of this ticket should be changed to better reflect the source of the issue.

I am not allowed to change anything related to the question or subject.

suggest a new title and I will change it afterwards.

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Giovanni Manghi wrote:

Casper Børgesen wrote:

Giovanni Manghi wrote:

... Anyway, if this observation is right then the title and description of this ticket should be changed to better reflect the source of the issue.

I am not allowed to change anything related to the question or subject.

suggest a new title and I will change it afterwards.

Title: Slow performance when using OGR FileGDB driver to read file geodatabases.

Description:
When loading file geodatabases into QGIS there is a big performance difference between using the OpenFileGDB driver and the OGR FileGDB driver. Opening geodatabases using the OpenFileGDB driver is almost instantaneous where using the OGR FileGDB driver seems to spend a lot of time opening and closing @.lock@ files during the import/removal operations.

• It affects QGIS 2.18, 3.2 and MASTER.
• It seems to be unaffected by whether any ESRI tools are installed or not.
• It might not affect Linux systems.

It might be related to issue #26232 where Even Rouault did some optimizations.

Steps to reproduce (https://www.youtube.com/watch?v=hvcPVObfBFA):
The OGR FileGDB driver needs to be installed in order to reproduce and the testdata_2.gdb.zip can be used in the test.

1. I have a FileGDB which I drag into QGIS (14 seconds).
2. I select all layers and press OK (7 seconds).
3. I remove all the layers (empty layer list and this process is slow too).
4. I drag the same FileGDB into QGIS again (14 seconds).
5. I select all layers and press OK (I was disturbed by a colleague after about 2.5 minutes and all layers was loaded when I came back after 15 minutes).

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

Title: Slow performance when using OGR FileGDB driver to read file geodatabases.

should not reflect the finding from here?
it happens, but only in some circumstance, correct?

[qgib]

Author Name: Casper Børgesen (Casper Børgesen)

---

Giovanni Manghi wrote:

Title: Slow performance when using OGR FileGDB driver to read file geodatabases.

should not reflect the finding from here?
it happens, but only in some circumstance, correct?

If I compare to using the OpenFileGDB driver, then the performance is slower (not as instantaneous as the OpenFileGDB driver) all the time - no matter if I use 2.18, 3.2 or MASTER. The issue seems to be much more severe the second time I load the data set, which is what I try to show in the video.

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

If I compare to using the OpenFileGDB driver, then the performance is slower (not as instantaneous as the OpenFileGDB driver) all the time - no matter if I use 2.18, 3.2 or MASTER. The issue seems to be much more severe the second time I load the data set, which is what I try to show in the video.

so you don't confirm only when the layers are added in the ways as stated here?

#27790 (comment) (edited)

from that comment I deduce that if the layers are added using the QGIS Browser the issue does not show(?).

[qgib]

Author Name: Andrea Giudiceandrea (@agiudiceandrea)

---

Giovanni Manghi wrote:

from that comment I deduce that if the layers are added using the QGIS Browser the issue does not show(?).

With the GDAL/OGR FileGDB driver plugin installed, using the Browser panel to add the layers to the map it seems that the second time is not slower then the first time: they are almost equally slow (it's a matter of minutes for add or remove the layers). Without the FileGDB driver plugin, each task takes just a few seconds.

[Pedro-Murteira]

I can't seem to replicate this issue anymore, I can drag the layers onto QGIS without any issue, it takes 1/2 seconds replicating each step mentioned in the ticket. Also I did not use any plugins, just a fresh profile. (QGIS 3.16,14 and 3.22.1., tested on Windows 10).

[github-actions]

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale".
If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue.
In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue.
If there is no further activity on this issue, it will be closed in a week.

[github-actions]

While we hate to see this happen, this issue has been automatically closed because it has not had any activity in the last 42 days despite being marked as feedback. If this issue should be reconsidered, please follow the guidelines in the previous comment and reopen this issue.
Or, if you have any further questions, there are also further support channels that can help you.