ci testsuite: TestQgsOfflineEditing fails with gdal >= 3.4.2 #48012
Labels
Bug
Either a bug report, or a bug fix. Let's hope for the latter!
Offline editing
testsuite
Issue related to testsuite
Comments
|
@nyalldawson do you think this changed result |
|
@Pedro-Murteira wanna add labels testsuite + upstream? |
|
That should be @nirvn |
t0b3
changed the title
Pedro-Murteira
added
testsuite
Done |
|
bisecting shows it is triggered by OSGeo/gdal@419b112 |
Pedro-Murteira
removed
the
Upstream
|
Checked it, concerning the implementation @m-kuhn mentioned with the QGIS/tests/src/core/testqgsofflineediting.cpp Line 214 in 20dfa3c |
rouault
added a commit
to rouault/QGIS
that referenced
this issue
Apr 2, 2022
Fixes qgis#48012 Basically, for offline editing, we ask QgsOgrProvider to adopt the behaviour it had before qgis#47098, that is use SQLite3 journal_mode = WAL even in read-only mode. Sorry, this "fix" is horrible, but I couldn't get with something better after many hours of investigation. The root cause seems to be that QgsOfflineEditing creates layers using direct OGR API or sqlite3 API, and also uses QgsOgrProviderUtils which has a GDAL dataset reuse mechanism. And all that does not place nicely at all. Maybe the root root cause is that the OGR GPKG driver establishes the list of layers at its opening, and if creates a new one behind its back, which is prone to happen with the dataset caching metchanism, the GDALDataset::GetLayerByName() method doesn't see the newly created layer. With that in mind, I tried to add calls to QgsOgrProviderUtils::invalidateCachedDatasets() but to no avail. Also QgsOgrProviderUtils::invalidateCachedLastModifiedDate() which is used by QgsOgrProvider::createEmptyLayer(), but to no avail. So basically I don't understand why offline editing worked before with a GPKG backend. I'd say it is pure luck. Note: I also saw that qgsofflineediting.cpp at line 315 calls offlineLayer->dataProvider()->invalidateConnections() but that method is only implemented in the Spatialite provider. I tried to implement it in the OGR one, but to no avail too.
rouault
added a commit
to rouault/QGIS
that referenced
this issue
Apr 2, 2022
Fixes qgis#48012 Basically, for offline editing, we ask QgsOgrProvider to adopt the behaviour it had before qgis#47098, that is use SQLite3 journal_mode = WAL even in read-only mode. Sorry, this "fix" is horrible, but I couldn't get with something better after many hours of investigation. The root cause seems to be that QgsOfflineEditing creates layers using direct OGR API or sqlite3 API, and also uses QgsOgrProvider which has a GDAL dataset reuse mechanism. And all that does not place nicely at all. Maybe the root root cause is that the OGR GPKG driver establishes the list of layers at its opening, and if creates a new one behind its back, which is prone to happen with the dataset caching metchanism, the GDALDataset::GetLayerByName() method doesn't see the newly created layer. With that in mind, I tried to add calls to QgsOgrProviderUtils::invalidateCachedDatasets() but to no avail. Also QgsOgrProviderUtils::invalidateCachedLastModifiedDate() which is used by QgsOgrProvider::createEmptyLayer(), but to no avail. So basically I don't understand why offline editing worked before with a GPKG backend. I'd say it is pure luck. Note: I also saw that qgsofflineediting.cpp at line 315 calls offlineLayer->dataProvider()->invalidateConnections() but that method is only implemented in the Spatialite provider. I tried to implement it in the OGR one, but to no avail too.
|
Horrible fix queued in #48059 |
t0b3
added a commit
to t0b3/QGIS
that referenced
this issue
Apr 3, 2022
Basically, for offline editing, we ask QgsOgrProvider to adopt the behaviour it had before qgis#47098, that is use SQLite3 journal_mode = WAL even in read-only mode. Sorry, this "fix" is horrible, but I couldn't get with something better after many hours of investigation. The root cause seems to be that QgsOfflineEditing creates layers using direct OGR API or sqlite3 API, and also uses QgsOgrProvider which has a GDAL dataset reuse mechanism. And all that does not place nicely at all. Maybe the root root cause is that the OGR GPKG driver establishes the list of layers at its opening, and if creates a new one behind its back, which is prone to happen with the dataset caching metchanism, the GDALDataset::GetLayerByName() method doesn't see the newly created layer. With that in mind, I tried to add calls to QgsOgrProviderUtils::invalidateCachedDatasets() but to no avail. Also QgsOgrProviderUtils::invalidateCachedLastModifiedDate() which is used by QgsOgrProvider::createEmptyLayer(), but to no avail. So basically I don't understand why offline editing worked before with a GPKG backend. I'd say it is pure luck. Note: I also saw that qgsofflineediting.cpp at line 315 calls offlineLayer->dataProvider()->invalidateConnections() but that method is only implemented in the Spatialite provider. I tried to implement it in the OGR one, but to no avail too.
This was referenced Apr 3, 2022
nyalldawson
pushed a commit
that referenced
this issue
Apr 4, 2022
Fixes #48012 Basically, for offline editing, we ask QgsOgrProvider to adopt the behaviour it had before #47098, that is use SQLite3 journal_mode = WAL even in read-only mode. Sorry, this "fix" is horrible, but I couldn't get with something better after many hours of investigation. The root cause seems to be that QgsOfflineEditing creates layers using direct OGR API or sqlite3 API, and also uses QgsOgrProvider which has a GDAL dataset reuse mechanism. And all that does not place nicely at all. Maybe the root root cause is that the OGR GPKG driver establishes the list of layers at its opening, and if creates a new one behind its back, which is prone to happen with the dataset caching metchanism, the GDALDataset::GetLayerByName() method doesn't see the newly created layer. With that in mind, I tried to add calls to QgsOgrProviderUtils::invalidateCachedDatasets() but to no avail. Also QgsOgrProviderUtils::invalidateCachedLastModifiedDate() which is used by QgsOgrProvider::createEmptyLayer(), but to no avail. So basically I don't understand why offline editing worked before with a GPKG backend. I'd say it is pure luck. Note: I also saw that qgsofflineediting.cpp at line 315 calls offlineLayer->dataProvider()->invalidateConnections() but that method is only implemented in the Spatialite provider. I tried to implement it in the OGR one, but to no avail too.
qgis-bot
pushed a commit
that referenced
this issue
Apr 4, 2022
Fixes #48012 Basically, for offline editing, we ask QgsOgrProvider to adopt the behaviour it had before #47098, that is use SQLite3 journal_mode = WAL even in read-only mode. Sorry, this "fix" is horrible, but I couldn't get with something better after many hours of investigation. The root cause seems to be that QgsOfflineEditing creates layers using direct OGR API or sqlite3 API, and also uses QgsOgrProvider which has a GDAL dataset reuse mechanism. And all that does not place nicely at all. Maybe the root root cause is that the OGR GPKG driver establishes the list of layers at its opening, and if creates a new one behind its back, which is prone to happen with the dataset caching metchanism, the GDALDataset::GetLayerByName() method doesn't see the newly created layer. With that in mind, I tried to add calls to QgsOgrProviderUtils::invalidateCachedDatasets() but to no avail. Also QgsOgrProviderUtils::invalidateCachedLastModifiedDate() which is used by QgsOgrProvider::createEmptyLayer(), but to no avail. So basically I don't understand why offline editing worked before with a GPKG backend. I'd say it is pure luck. Note: I also saw that qgsofflineediting.cpp at line 315 calls offlineLayer->dataProvider()->invalidateConnections() but that method is only implemented in the Spatialite provider. I tried to implement it in the OGR one, but to no avail too.
qgis-bot
pushed a commit
that referenced
this issue
Apr 4, 2022
Fixes #48012 Basically, for offline editing, we ask QgsOgrProvider to adopt the behaviour it had before #47098, that is use SQLite3 journal_mode = WAL even in read-only mode. Sorry, this "fix" is horrible, but I couldn't get with something better after many hours of investigation. The root cause seems to be that QgsOfflineEditing creates layers using direct OGR API or sqlite3 API, and also uses QgsOgrProvider which has a GDAL dataset reuse mechanism. And all that does not place nicely at all. Maybe the root root cause is that the OGR GPKG driver establishes the list of layers at its opening, and if creates a new one behind its back, which is prone to happen with the dataset caching metchanism, the GDALDataset::GetLayerByName() method doesn't see the newly created layer. With that in mind, I tried to add calls to QgsOgrProviderUtils::invalidateCachedDatasets() but to no avail. Also QgsOgrProviderUtils::invalidateCachedLastModifiedDate() which is used by QgsOgrProvider::createEmptyLayer(), but to no avail. So basically I don't understand why offline editing worked before with a GPKG backend. I'd say it is pure luck. Note: I also saw that qgsofflineediting.cpp at line 315 calls offlineLayer->dataProvider()->invalidateConnections() but that method is only implemented in the Spatialite provider. I tried to implement it in the OGR one, but to no avail too.
nyalldawson
pushed a commit
that referenced
this issue
Apr 4, 2022
Fixes #48012 Basically, for offline editing, we ask QgsOgrProvider to adopt the behaviour it had before #47098, that is use SQLite3 journal_mode = WAL even in read-only mode. Sorry, this "fix" is horrible, but I couldn't get with something better after many hours of investigation. The root cause seems to be that QgsOfflineEditing creates layers using direct OGR API or sqlite3 API, and also uses QgsOgrProvider which has a GDAL dataset reuse mechanism. And all that does not place nicely at all. Maybe the root root cause is that the OGR GPKG driver establishes the list of layers at its opening, and if creates a new one behind its back, which is prone to happen with the dataset caching metchanism, the GDALDataset::GetLayerByName() method doesn't see the newly created layer. With that in mind, I tried to add calls to QgsOgrProviderUtils::invalidateCachedDatasets() but to no avail. Also QgsOgrProviderUtils::invalidateCachedLastModifiedDate() which is used by QgsOgrProvider::createEmptyLayer(), but to no avail. So basically I don't understand why offline editing worked before with a GPKG backend. I'd say it is pure luck. Note: I also saw that qgsofflineediting.cpp at line 315 calls offlineLayer->dataProvider()->invalidateConnections() but that method is only implemented in the Spatialite provider. I tried to implement it in the OGR one, but to no avail too.
nyalldawson
pushed a commit
that referenced
this issue
Apr 4, 2022
Fixes #48012 Basically, for offline editing, we ask QgsOgrProvider to adopt the behaviour it had before #47098, that is use SQLite3 journal_mode = WAL even in read-only mode. Sorry, this "fix" is horrible, but I couldn't get with something better after many hours of investigation. The root cause seems to be that QgsOfflineEditing creates layers using direct OGR API or sqlite3 API, and also uses QgsOgrProvider which has a GDAL dataset reuse mechanism. And all that does not place nicely at all. Maybe the root root cause is that the OGR GPKG driver establishes the list of layers at its opening, and if creates a new one behind its back, which is prone to happen with the dataset caching metchanism, the GDALDataset::GetLayerByName() method doesn't see the newly created layer. With that in mind, I tried to add calls to QgsOgrProviderUtils::invalidateCachedDatasets() but to no avail. Also QgsOgrProviderUtils::invalidateCachedLastModifiedDate() which is used by QgsOgrProvider::createEmptyLayer(), but to no avail. So basically I don't understand why offline editing worked before with a GPKG backend. I'd say it is pure luck. Note: I also saw that qgsofflineediting.cpp at line 315 calls offlineLayer->dataProvider()->invalidateConnections() but that method is only implemented in the Spatialite provider. I tried to implement it in the OGR one, but to no avail too.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What is the bug or the crash?
test TestQgsOfflineEditing fails with gdal 3.4.2 related to fe410ec
Steps to reproduce the issue
compile & test master with gdal 3.4.2 (i.e. on fedora 36 as on PR #47887)
details see https://cdash.orfeo-toolbox.org/testDetails.php?test=35806854&build=95038
Versions
master with gdal=3.4.2
Supported QGIS version
New profile
Additional context
QGIS/tests/src/core/testqgsofflineediting.cpp
Lines 296 to 299 in 20dfa3c
The text was updated successfully, but these errors were encountered: