Inefficient use of GDAL block cache during raster rendering

[qgib]

Author Name: Even Rouault (@rouault)
Original Redmine Issue: 16006
Affected QGIS version: 2.8.0
Redmine category:rasters

---

Each time the canvas is rendered, the rendering of GDAL raster layers is done with a new GDAL dataset object. Which means the dataset must be re-opened (potentially source of slowness), but more importantly, the GDAL block cache is of no use between 2 canvas rendering. For example when panning in the same area, cached tiles could potentially be reused. One can especially observe the slowness with slow compression methods (JPEG2000, LZMA, etc)
This issue didn't exist in ancient QGIS version (it exists in 2.8 and ongoing, perhaps older). This is due to the cloning of the QgsRasterPipe object at rendering.
Presumably some pool of GDAL dataset objects should be used to avoid those constant open & close operations

[qgib]

Author Name: Giovanni Manghi (@gioman)

---

• easy_fix was configured as 0
• regression was configured as 0

[qgib]

Author Name: Even Rouault (@rouault)

---

Just hit that issue again with a GDAL driver that does network accesses. Each time you scroll the canvas, the previously cached blocks are discarded.

For reference, stack trace is:

(gdb) bt
#0 GDALOpen (pszFilename=0x537abc8 "/home/even/qgis/2.18/build/eedai", eAccess=GA_ReadOnly) at gdaldataset.cpp:2568
#_1 0x00007fa16a38a908 in QgsGdalProviderBase::gdalOpen (pszFilename=0x537abc8 "/home/even/qgis/2.18/build/eedai", eAccess=GA_ReadOnly) at /home/even/qgis/2.18/src/providers/gdal/qgsgdalproviderbase.cpp:272
#_2 0x00007fa16a38c7d7 in QgsGdalProvider::QgsGdalProvider (this=0x52fd2b0, uri=..., update=false) at /home/even/qgis/2.18/src/providers/gdal/qgsgdalprovider.cpp:177
#_3 0x00007fa16a38cca9 in QgsGdalProvider::clone (this=0x555d4d0) at /home/even/qgis/2.18/src/providers/gdal/qgsgdalprovider.cpp:192
#_4 0x00007fa16a38cd31 in covariant return thunk to QgsGdalProvider::clone() const () at /home/even/qgis/2.18/src/providers/gdal/qgsgdalprovider.cpp:195
#_5 0x00007fa1c5d93e4d in QgsRasterPipe::QgsRasterPipe (this=0x54a9dd0, thePipe=...) at /home/even/qgis/2.18/src/core/raster/qgsrasterpipe.cpp:34
#_6 0x00007fa1c5d917fe in QgsRasterLayerRenderer::QgsRasterLayerRenderer (this=0x537aed0, layer=0x555d2c0, rendererContext=...) at /home/even/qgis/2.18/src/core/raster/qgsrasterlayerrenderer.cpp:176
#_7 0x00007fa1c5d84a65 in QgsRasterLayer::createMapRenderer (this=0x555d2c0, rendererContext=...) at /home/even/qgis/2.18/src/core/raster/qgsrasterlayer.cpp:258
#_8 0x00007fa1c5ad6ec3 in QgsMapRendererJob::prepareJobs (this=0x45a0d50, painter=0x45a9cf0, labelingEngine=0x0, labelingEngine2=0x447c580) at /home/even/qgis/2.18/src/core/qgsmaprendererjob.cpp:302
#_9 0x00007fa1c5ad1320 in QgsMapRendererCustomPainterJob::start (this=0x45a0d50) at /home/even/qgis/2.18/src/core/qgsmaprenderercustompainterjob.cpp:100
#_10 0x00007fa1c5add8b4 in QgsMapRendererSequentialJob::start (this=0x563a210) at /home/even/qgis/2.18/src/core/qgsmaprenderersequentialjob.cpp:74
#_11 0x00007fa1c6840206 in QgsMapCanvas::refreshMap (this=0x2b38200) at /home/even/qgis/2.18/src/gui/qgsmapcanvas.cpp:738
#_12 0x00007fa1c6920114 in QgsMapCanvas::qt_static_metacall (_o=0x2b38200, _c=QMetaObject::InvokeMetaMethod, _id=49, _a=0x7ffc3945a200) at /home/even/qgis/2.18/build/src/gui/moc_qgsmapcanvas.cxx:191
#_13 0x00007fa1c52a4ddf in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_14 0x00007fa1c52accdf in QSingleShotTimer::timerEvent(QTimerEvent*) () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_15 0x00007fa1c52a40ec in QObject::event(QEvent*) () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_16 0x00007fa1c4622e3c in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /home/even/install-qt-4.8.5/lib/libQtGui.so.4
#_17 0x00007fa1c46272ba in QApplication::notify(QObject*, QEvent*) () from /home/even/install-qt-4.8.5/lib/libQtGui.so.4
#_18 0x00007fa1c5978edf in QgsApplication::notify (this=0x7ffc3945b580, receiver=0x57a49a0, event=0x7ffc3945a960) at /home/even/qgis/2.18/src/core/qgsapplication.cpp:281
#_19 0x00007fa1c528f9ee in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_20 0x00007fa1c52c0712 in QTimerInfoList::activateTimers() () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_21 0x00007fa1c52bd704 in timerSourceDispatch(_GSource*, int ()(void), void*) () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_22 0x00007fa1bc21c197 in g_main_context_dispatch () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#_23 0x00007fa1bc21c3f0 in ?? () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#_24 0x00007fa1bc21c49c in g_main_context_iteration () from /lib/x86_64-linux-gnu/libglib-2.0.so.0
#_25 0x00007fa1c52bdf16 in QEventDispatcherGlib::processEvents(QFlagsQEventLoop::ProcessEventsFlag) () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_26 0x00007fa1c46c319e in QGuiEventDispatcherGlib::processEvents(QFlagsQEventLoop::ProcessEventsFlag) () from /home/even/install-qt-4.8.5/lib/libQtGui.so.4
#_27 0x00007fa1c528e69f in QEventLoop::processEvents(QFlagsQEventLoop::ProcessEventsFlag) () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_28 0x00007fa1c528e928 in QEventLoop::exec(QFlagsQEventLoop::ProcessEventsFlag) () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_29 0x00007fa1c5293668 in QCoreApplication::exec() () from /home/even/install-qt-4.8.5/lib/libQtCore.so.4
#_30 0x000000000040ae18 in main (argc=2, argv=0x7ffc3945b9e8) at /home/even/qgis/2.18/src/app/main.cpp:1287

---

• description was changed from Each time the canvas is rendered, the rendering of GDAL raster layers is done with a new GDAL dataset object. Which means the dataset must be re-opened (potentially source of slowness), but more importantly, the GDAL block cache is of no use between 2 canvas rendering. For example when panning in the same area, cached tiles could potentially be reused. One can especially observe the slowness with slow compression methods (JPEG2000, LZMA, etc)
  This issue didn't exist in ancient QGIS version (it exists in 2.8 and ongoing, perhaps older). This is due to the cloning of the QgsRasterPipe object at rendering.
  Presumably some pool of GDAL dataset objects should be used to avoid those constant open & close operations to Each time the canvas is rendered, the rendering of GDAL raster layers is done with a new GDAL dataset object. Which means the dataset must be re-opened (potentially source of slowness), but more importantly, the GDAL block cache is of no use between 2 canvas rendering. For example when panning in the same area, cached tiles could potentially be reused. One can especially observe the slowness with slow compression methods (JPEG2000, LZMA, etc)
  This issue didn't exist in ancient QGIS version (it exists in 2.8 and ongoing, perhaps older). This is due to the cloning of the QgsRasterPipe object at rendering.
  Presumably some pool of GDAL dataset objects should be used to avoid those constant open & close operations

[qgib]

Author Name: Even Rouault (@rouault)

---

Applied in changeset 3b2b98b.

---

• done_ratio was changed from 0 to 100
• status_id was changed from Open to Closed