Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash with VRT layers using Python pixel functions #41048

Closed
letmaik opened this issue Jan 17, 2021 · 13 comments · Fixed by OSGeo/gdal#3424
Closed

Crash with VRT layers using Python pixel functions #41048

letmaik opened this issue Jan 17, 2021 · 13 comments · Fixed by OSGeo/gdal#3424
Assignees
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Crash/Data Corruption

Comments

@letmaik
Copy link
Contributor

letmaik commented Jan 17, 2021

User Feedback

Originally reported at GIS4WRF/gis4wrf#202.
The crash happens when showing a VRT layer with Python-backed pixel function.
The same code works fine in 3.12 and fails in 3.16, I haven't tested versions in between.

The VRT is (simply save to file test.vrt and drag and drop into QGIS):

<VRTDataset rasterXSize="36" rasterYSize="13">
  <SRS dataAxisToSRSAxisMapping="1,2">GEOGCS["unknown",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Longitude",EAST],AXIS["Latitude",NORTH]]</SRS>
  <GeoTransform> -4.5871999999999993e+00,  1.0000000000000000e+00,  0.0000000000000000e+00,  3.3713099999999997e+01,  0.0000000000000000e+00, -1.0000000000000000e+00</GeoTransform>
  <VRTRasterBand band="1" dataType="Byte" subClass="VRTDerivedRasterBand">
    <PixelFunctionLanguage>Python</PixelFunctionLanguage>
    <PixelFunctionType>gdal_checkerboard_pixelfunction</PixelFunctionType>
            <PixelFunctionCode><![CDATA[
def gdal_checkerboard_pixelfunction(in_ar, out_ar, xoff, yoff, xsize, ysize, raster_xsize,
                                    raster_ysize, buf_radius, gt):
    x_even = xoff % 2 == 0
    y_even = yoff % 2 == 0
    is_white = (x_even and y_even) or (not x_even and not y_even)
    out_ar[:] = 1 - is_white
    out_ar[::2, 1::2] = is_white
    out_ar[1::2, ::2] = is_white
            ]]>
        </PixelFunctionCode>
  </VRTRasterBand>
</VRTDataset>

Apparently somewhere along the line a Python exception is raised (see native stack trace below) but the exception is not caught in QGIS and displayed, but rather lets it crash, which makes diagnosing the issue hard.

Report Details

Crash ID: 081f51e990863dc1c9591f81f41f04f70cdad7e9

Stack Trace

PyModule_Create2 :
PyInit__ctypes :
PyUnicode_EncodeCharmap :
PyUnicode_EncodeCharmap :
PyUnicode_EncodeCharmap :
PyMethodDef_RawFastCallDict :
PyException_SetTraceback :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyFunction_FastCallDict :
PyObject_CallMethodIdObjArgs :
PyObject_CallMethodIdObjArgs :
PyUnicode_FindChar :
PyImport_ImportModuleLevelObject :
PyType_Lookup :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyImport_Import :
PyImport_Import :
PyMethodDef_RawFastCallDict :
PyException_SetTraceback :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyFunction_FastCallDict :
PyObject_CallMethodIdObjArgs :
PyObject_CallMethodIdObjArgs :
PyUnicode_FindChar :
PyImport_ImportModuleLevelObject :
PyType_Lookup :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyImport_Import :
PyImport_Import :
PyMethodDef_RawFastCallDict :
PyException_SetTraceback :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyFunction_FastCallDict :
PyObject_CallMethodIdObjArgs :
PyObject_CallMethodIdObjArgs :
PyUnicode_FindChar :
PyImport_ImportModuleLevelObject :
PyDict_GetItemString :
PyException_SetTraceback :
PyException_SetTraceback :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyFunction_FastCallDict :
PyObject_CallMethodIdObjArgs :
PyObject_CallMethodIdObjArgs :
PyImport_ImportModuleLevelObject :
PyType_Lookup :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyImport_Import :
PyImport_Import :
PyMethodDef_RawFastCallDict :
PyException_SetTraceback :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyFunction_FastCallDict :
PyObject_CallMethodIdObjArgs :
PyObject_CallMethodIdObjArgs :
PyUnicode_FindChar :
PyImport_ImportModuleLevelObject :
PyType_Lookup :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyEval_EvalCodeEx :
PyEval_EvalCode :
PyImport_ImportFrozenModuleObject :
PyImport_ExecCodeModuleObject :
PyImport_ExecCodeModuleWithPathnames :
VRTDerivedRasterBand::InitializePython :
VRTDerivedRasterBand::IRasterIO :
VRTSourcedRasterBand::IReadBlock :
GDALRasterBand::GetLockedBlockRef :
GDALRasterBand::ComputeStatistics :
GDALRasterBand::GetStatistics :
QgsGdalProvider::bandStatistics qgsgdalprovider.cpp:2787
QgsRasterLayer::computeMinMax qgsrasterlayer.cpp:981
QgsRasterLayer::setContrastEnhancement qgsrasterlayer.cpp:1093
QgsRasterLayer::setDefaultContrastEnhancement qgsrasterlayer.cpp:1413
QgsRasterLayer::setDataSource qgsrasterlayer.cpp:954
QgsRasterLayer::QgsRasterLayer qgsrasterlayer.cpp:132
sipQgsRasterLayer::sipQgsRasterLayer sip_corepart3.cpp:14006
init_type_QgsRasterLayer sip_corepart3.cpp:18849
PyObject_FastCallKeywords :
PyObject_FastCallKeywords :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyEval_EvalCodeWithName :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyMethodDef_RawFastCallKeywords :
PyEval_EvalFrameDefault :
PyFunction_FastCallDict :
PyMethodDef_RawFastCallDict :
PyObject_Call :
PyInit_QtCore :
PyInit_QtCore :
PyInit_QtCore :
PyInit_QtCore :
PyInit_QtCore :
QMetaObject::activate :
QAbstractButton::clicked :
QAbstractButton::click :
QAbstractButton::mouseReleaseEvent :
QWidget::event :
QApplicationPrivate::notify_helper :
QApplication::notify :
QgsApplication::notify qgsapplication.cpp:458
QCoreApplication::notifyInternal2 :
QApplicationPrivate::sendMouseEvent :
QSizePolicy::QSizePolicy :
QSizePolicy::QSizePolicy :
QApplicationPrivate::notify_helper :
QApplication::notify :
QgsApplication::notify qgsapplication.cpp:458
QCoreApplication::notifyInternal2 :
QGuiApplicationPrivate::processMouseEvent :
QWindowSystemInterface::sendWindowSystemEvents :
QEventDispatcherWin32::processEvents :
UserCallWinProcCheckWow :
DispatchMessageWorker :
QEventDispatcherWin32::processEvents :
qt_plugin_query_metadata :
QEventLoop::exec :
QCoreApplication::exec :
main main.cpp:1637
BaseThreadInitThunk :
RtlUserThreadStart :

QGIS Info
QGIS Version: 3.16.2-Hannover
QGIS code revision: f1660f9
Compiled against Qt: 5.11.2
Running against Qt: 5.11.2
Compiled against GDAL: 3.1.4
Running against GDAL: 3.1.4

System Info
CPU Type: x86_64
Kernel Type: winnt
Kernel Version: 10.0.19041

@letmaik letmaik added the Bug Either a bug report, or a bug fix. Let's hope for the latter! label Jan 17, 2021
@roya0045
Copy link
Contributor

Are those duplicate calls? A typical python stack trace would be more helpful here. And VRTDerivedRasterBand is a GDAL construct. I'm not sure if the issue if the handling of python in gdal or qgis.

@roya0045
Copy link
Contributor

Maybe @rouault would have a clue.

@elpaso
Copy link
Contributor

elpaso commented Jan 18, 2021

Tested on Linux master, no crashes here:

immagine

@elpaso elpaso added the Feedback Waiting on the submitter for answers label Jan 18, 2021
@letmaik
Copy link
Contributor Author

letmaik commented Jan 19, 2021

@elpaso Which gdal version are you using?

@elpaso
Copy link
Contributor

elpaso commented Jan 19, 2021

@letmaik Running against GDAL/OGR | 3.3.0dev-c3f04b3163

@rouault
Copy link
Contributor

rouault commented Jan 23, 2021

@letmaik Can you reproduce that on the command line with gdal_translate for example ?

@letmaik
Copy link
Contributor Author

letmaik commented Jan 23, 2021

@rouault No I can't, it works there.

>"C:\Program Files\QGIS 3.16\bin\gdal_translate.exe" --config GDAL_VRT_ENABLE_PYTHON YES --config CPL_DEBUG ON test.vrt test.tif
GDAL: Auto register C:\Program Files\QGIS 3.16\bin\gdalplugins\gdal_ECW_JP2ECW.dll using GDALRegister_ECW_JP2ECW.
GDAL: Auto register C:\Program Files\QGIS 3.16\bin\gdalplugins\gdal_GEOR.dll using GDALRegister_GEOR.
GDAL: Auto register C:\Program Files\QGIS 3.16\bin\gdalplugins\gdal_HDF5.dll using GDALRegister_HDF5.
GDAL: Auto register C:\Program Files\QGIS 3.16\bin\gdalplugins\gdal_MG4Lidar.dll using GDALRegister_MG4Lidar.
GDAL: Auto register C:\Program Files\QGIS 3.16\bin\gdalplugins\gdal_MrSID.dll using GDALRegister_MrSID.
GDAL: Auto register C:\Program Files\QGIS 3.16\bin\gdalplugins\ogr_OCI.dll using RegisterOGROCI.
GDAL: Auto register C:\Program Files\QGIS 3.16\bin\gdalplugins\ogr_SOSI.dll using RegisterOGRSOSI.
GDAL: GDALOpen(test.vrt, this=00000241E64EAA70) succeeds as VRT.
GDAL: Using GTiff driver
Input file size is 36, 13
GDAL: Using GTiff driver
GDAL: QuietDelete(test.tif) invoking Delete()
GDAL: GDALOpen(test.tif, this=00000241E64D5410) succeeds as GTiff.
GDAL: GDALDefaultOverviews::OverviewScan()
GDAL: GDALClose(test.tif, this=00000241E64D5410)
GDAL: GDALDefaultOverviews::OverviewScan()
0GDAL: GDAL_CACHEMAX = 816 MB
GDAL: GDALDatasetCopyWholeRaster(): 36*13 swaths, bInterleave=0
GDAL: Found C:\Program Files\QGIS 3.16\apps\Python37\python.exe
GDAL: C:\Program Files\QGIS 3.16\apps\Python37\python37.dll loaded
GDAL: Python version used: 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]
GDAL: Py_Initialize()
...10...20...30...40...50...60...70...80...90...100 - done.
GDAL: Flushing dirty blocks: 0...10...20...30...40...50...60...70...80...90...100 - done.
GDAL: GDALClose(test.tif, this=00000241E64D5410)
GDAL: GDALClose(test.vrt, this=00000241E64EAA70)
GDAL: In GDALDestroy - unloading GDAL shared library.

Setting CPL_DEBUG=ON and CPL_LOG=path/to/log.txt before starting QGIS it gets only that far:

GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_ECW_JP2ECW.dll using GDALRegister_ECW_JP2ECW.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_GEOR.dll using GDALRegister_GEOR.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_HDF5.dll using GDALRegister_HDF5.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_MG4Lidar.dll using GDALRegister_MG4Lidar.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_MrSID.dll using GDALRegister_MrSID.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\ogr_OCI.dll using RegisterOGROCI.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\ogr_SOSI.dll using RegisterOGRSOSI.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_ECW_JP2ECW.dll using GDALRegister_ECW_JP2ECW.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_GEOR.dll using GDALRegister_GEOR.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_HDF5.dll using GDALRegister_HDF5.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_MG4Lidar.dll using GDALRegister_MG4Lidar.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_MrSID.dll using GDALRegister_MrSID.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\ogr_OCI.dll using RegisterOGROCI.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\ogr_SOSI.dll using RegisterOGRSOSI.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_ECW_JP2ECW.dll using GDALRegister_ECW_JP2ECW.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_GEOR.dll using GDALRegister_GEOR.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_HDF5.dll using GDALRegister_HDF5.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_MG4Lidar.dll using GDALRegister_MG4Lidar.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_MrSID.dll using GDALRegister_MrSID.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\ogr_OCI.dll using RegisterOGROCI.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\ogr_SOSI.dll using RegisterOGRSOSI.
GDAL: GDALOpen(C:/Users/maik/Desktop/test.vrt, this=0000017F22009F80) succeeds as VRT.
GDAL: GDALClose(C:/Users/maik/Desktop/test.vrt, this=0000017F22009F80)
GDAL: GDALOpen(C:/Users/maik/Desktop/test.vrt, this=0000017F22009F80) succeeds as VRT.
GDAL: Computing area of interest: -4.5872, 20.7131, 31.4128, 33.7131
GDAL: GDALDefaultOverviews::OverviewScan()
GDAL: GDAL_CACHEMAX = 816 MB
GDAL: Found C:\PROGRA~1\QGIS3~1.16\apps\Python37\python.exe
GDAL: C:\PROGRA~1\QGIS3~1.16\apps\Python37\python37.dll loaded
GDAL: Python version used: 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]
GDAL: Py_Initialize()

Note that 3 users can reproduce this on Windows in QGIS, including myself.

The fact that the log is likely incomplete and I don't know how to get the actual error message is really frustrating. I have a feeling that more is dumped to stdout (probably some Python exception or similar) but since there's no console window for QGIS it gets lost.

@rouault
Copy link
Contributor

rouault commented Jan 23, 2021

GDAL: Found C:\PROGRA1\QGIS31.16\apps\Python37\python.exe
GDAL: C:\PROGRA1\QGIS31.16\apps\Python37\python37.dll loaded

Hum, those lines in the QGIS log are the likely cause of the error. It means that for some reason GDAL fails to detect the Python interpretrer already loaded by QGIS, and thus it loads the python37.dll once more, hence a crash.

Could you try the following

set GDAL_VRT_ENABLE_PYTHON=YES
set CPL_DEBUG=ON
python -c "from osgeo import gdal; ds = gdal.Open('test.vrt'); print(ds.GetRasterBand(1).Checksum()"

@letmaik
Copy link
Contributor Author

letmaik commented Jan 23, 2021

Fwiw just tried the qgis-dev (compiled against gdal 3.1.4, running against 3.3.0dev) and qgis-rel-dev (compiled and running against gdal 3.3.0dev) variants via the OSGeo4W installer and get the same crash.

I'm not sure where you want me to run the python command from, I'm sure the below fails for other reasons.

"C:\PROGRA~1\QGIS3~1.16\apps\Python37\python.exe" -c "from osgeo import gdal; ds = gdal.Open('test.vrt'); print(ds.GetRasterBand(1).Checksum())"
Traceback (most recent call last):
  File "C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\site-packages\osgeo\__init__.py", line 18, in swig_import_helper
    return importlib.import_module(mname)
  File "C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 670, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 583, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1043, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\site-packages\osgeo\__init__.py", line 32, in <module>
    _gdal = swig_import_helper()
  File "C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\site-packages\osgeo\__init__.py", line 31, in swig_import_helper
    return importlib.import_module('_gdal')
  File "C:\PROGRA~1\QGIS3~1.16\apps\Python37\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_gdal'

@rouault
Copy link
Contributor

rouault commented Jan 23, 2021

I'm not sure where you want me to run the python command from

you probably also need to set PATH=C:\Program Files\QGIS 3.16\bin;%PATH%

@letmaik
Copy link
Contributor Author

letmaik commented Jan 24, 2021

GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_ECW_JP2ECW.dll using GDALRegister_ECW_JP2ECW.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_GEOR.dll using GDALRegister_GEOR.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_HDF5.dll using GDALRegister_HDF5.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_MG4Lidar.dll using GDALRegister_MG4Lidar.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\gdal_MrSID.dll using GDALRegister_MrSID.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\ogr_OCI.dll using RegisterOGROCI.
GDAL: Auto register C:\PROGRA~1\QGIS3~1.16\bin\gdalplugins\ogr_SOSI.dll using RegisterOGRSOSI.
GDAL: GDALOpen(test.vrt, this=000002473DD60920) succeeds as VRT.
GDAL: Current process has python symbols loaded
GDAL: Python version used: 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]
234
GDAL: GDALClose(test.vrt, this=000002473DD60920)
GDAL: In GDALDestroy - unloading GDAL shared library.

@fiedt
Copy link

fiedt commented Jan 31, 2021

This seems to be related to my issue #39962

@rouault
Copy link
Contributor

rouault commented Feb 2, 2021

Fixed in GDAL master and 3.2 branch (for GDAL 3.2.2)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Crash/Data Corruption
Projects
None yet
5 participants