initGUI() results in Segmentation fault on macOS in standalone PyQgis

[tillhainbach]

Describe the bug
When trying to use PyQgis for standalone python-qgis scripts or trying to run Unittest using qgis.testing, python/pyqgis crashes with segmentation fault: 11.

Also reported here:
https://stackoverflow.com/questions/64678135/segmentation-fault-11-when-using-initqgis-macos

Reproduced the error with QGIS for macOS 3.16.1 and nightly build.

How to Reproduce

in qgis_test_script.sh

# for Qgis 3.10 LTR pass "3.10" as $1
# Qgis PR version is stored under QGIS.app on macOS
QGIS_VERSION=$1

# store PATH and PYTHONPATH 
OLD_PATH=$PATH
OLD_PYTHONPATH=$PYTHONPATH

# set env variables:
export PATH=/Applications/QGIS$QGIS_VERSION.app/Contents/MacOS/bin:$PATH
export PYTHONPATH=/Applications/QGIS$QGIS_VERSION.app/Contents/Resources/python
export QT_QPA_PLATFORM_PLUGIN_PATH=/Applications/QGIS$QGIS_VERSION.app/Contents/PlugIns/platforms/

# check correct python bin is used
echo "Using python3 from $(which python3)"

# run start_app() from qgis.testing which uses initGUI(). Should print "Application state:..."
python3 -c "from qgis.testing import start_app; start_app()"

# restore and clean up
export PATH=$OLD_PATH
export PYTHONPATH=$OLD_PYTHONPATH
unset QT_QPA_PLATFORM_PLUGIN_PATH

failing example:

$ bash qgis_test_script.sh

this should work:

$ bash qgis_test_scrip.sh 3.10

QGIS and OS versions

macOS 10.15.7
QGIS: 3.16.1 and 3.17-master (nightly build)

[gioman]

Have you raised your issue also on the developers mailing list?

[tillhainbach]

Output from lldb:

(lldb) run test_qgis.py
Process 27773 launched: '/Applications/QGIS.app/Contents/MacOS/bin/python3' (x86_64)
Process 27773 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000000000000
error: memory read failed for 0x0
Target 0: (python3) stopped.
(lldb) tb
No breakpoints currently set.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000000000000
    frame #1: 0x00007fff69747ab4 libsqlite3.dylib`createFunctionApi + 148
    frame #2: 0x00007fff6967d22a libsqlite3.dylib`sqlite3_create_function_v2 + 42
    frame #3: 0x00000001124d5200 libspatialite.7.dylib`register_spatialite_sql_functions + 96
    frame #4: 0x000000011252121b libspatialite.7.dylib`spatialite_init_ex + 107
    frame #5: 0x000000010b1d7758 qgis_core`spatialite_database_unique_ptr::open_v2(QString const&, int, char const*) + 184
    frame #6: 0x000000010aef4999 qgis_core`QgsAuxiliaryStorage::createDB(QString const&) + 41
    frame #7: 0x000000010aef2a95 qgis_core`QgsAuxiliaryStorage::open(QString const&) + 373
    frame #8: 0x000000010aef2cfb qgis_core`QgsAuxiliaryStorage::QgsAuxiliaryStorage(QString const&, bool) + 91
    frame #9: 0x000000010b11fede qgis_core`QgsProject::QgsProject(QObject*) + 590
    frame #10: 0x000000010b124bca qgis_core`QgsProject::instance() + 42
    frame #11: 0x000000010aad6e77 qgis_core`QgsStyleModel::QgsStyleModel(QgsStyle*, QObject*) + 2119
    frame #12: 0x000000010aec018c qgis_core`QgsApplication::init(QString) + 4604
    frame #13: 0x000000010aecacea qgis_core`QgsApplication::initQgis() + 234
    frame #14: 0x0000000119156911 _core.so`meth_QgsApplication_initQgis(_object*, _object*) + 65
    frame #15: 0x0000000100130483 libpython3.7m.dylib`_PyMethodDef_RawFastCallKeywords + 387
    frame #16: 0x000000010012f8e6 libpython3.7m.dylib`_PyCFunction_FastCallKeywords + 38
    frame #17: 0x00000001001e3bd5 libpython3.7m.dylib`call_function + 213
    frame #18: 0x00000001001df0d2 libpython3.7m.dylib`_PyEval_EvalFrameDefault + 20882
    frame #19: 0x000000010012f825 libpython3.7m.dylib`_PyFunction_FastCallKeywords + 661
    frame #20: 0x00000001001e3b9f libpython3.7m.dylib`call_function + 159
    frame #21: 0x00000001001de6bb libpython3.7m.dylib`_PyEval_EvalFrameDefault + 18299
    frame #22: 0x00000001001e4263 libpython3.7m.dylib`_PyEval_EvalCodeWithName + 563
    frame #23: 0x00000001001d9eb0 libpython3.7m.dylib`PyEval_EvalCode + 48
    frame #24: 0x00000001002b943c libpython3.7m.dylib`PyRun_FileExFlags + 236
    frame #25: 0x00000001002b8b2a libpython3.7m.dylib`PyRun_SimpleFileExFlags + 522
    frame #26: 0x00000001002c6dfd libpython3.7m.dylib`pymain_run_file + 125
    frame #27: 0x00000001002c6953 libpython3.7m.dylib`pymain_run_filename + 83
    frame #28: 0x00000001002c4adb libpython3.7m.dylib`pymain_run_python + 107
    frame #29: 0x00000001002c46eb libpython3.7m.dylib`pymain_main + 27
    frame #30: 0x00000001002c4742 libpython3.7m.dylib`_Py_UnixMain + 50
    frame #31: 0x00007fff6a6b2cc9 libdyld.dylib`start + 1
    frame #32: 0x00007fff6a6b2cc9 libdyld.dylib`start + 1

# test_qgis.py
import qgis.testing import start_app

start_app()

[tillhainbach]

Have you raised your issue also on the developers mailing list?

No I did not. Should I do so and if yes where?

[gioman]

Should I do

@tillhainbach you'll reach a much wider audience that can tell you if there is anything wrong in your code

where

https://qgis.org/en/site/forusers/support.html#mailing-lists

[github-actions]

The QGIS project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 14 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

• that all unit tests are passing

• that all comments by reviewers have been addressed

• that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes

  • add a description of workflows which this pull request fixes

  • add screenshots if applicable

• that you have written unit tests where possible
  In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
  If there is no further activity on this pull request, it will be closed in a week.

[PeterPetrik]

the system sqlite is picked, but it is not compatible with the sqlite bundled with QGIS. The modified script should work

# for Qgis 3.10 LTR pass "3.10" as $1
# Qgis PR version is stored under QGIS.app on macOS
QGIS_VERSION=$1

# store PATH and PYTHONPATH 
OLD_PATH=$PATH
OLD_PYTHONPATH=$PYTHONPATH

# set env variables:
export PATH=/Applications/QGIS$QGIS_VERSION.app/Contents/MacOS/bin:$PATH
export PYTHONPATH=/Applications/QGIS$QGIS_VERSION.app/Contents/Resources/python
export QT_QPA_PLATFORM_PLUGIN_PATH=/Applications/QGIS$QGIS_VERSION.app/Contents/PlugIns/platforms/
export DYLD_INSERT_LIBRARIES=/Applications/QGIS$QGIS_VERSION.app/Contents/MacOS/lib/libsqlite3.dylib

# check correct python bin is used
echo "Using python3 from $(which python3)"

# run start_app() from qgis.testing which uses initGUI(). Should print "Application state:..."
python3 -c "from qgis.testing import start_app; start_app()"

# restore and clean up
export PATH=$OLD_PATH
export PYTHONPATH=$OLD_PYTHONPATH
unset QT_QPA_PLATFORM_PLUGIN_PATH
unset DYLD_INSERT_LIBRARIES

[tillhainbach]

Nice! Thank you! Should this be added to the documentation for future reference? I'll can open a PR if someone points me to the correct project-repo.

[PeterPetrik]

no sure if required for docs, maybe @DelazJ can shed a light?

[DelazJ]

I'm not sure if it has to be documented but when it comes to python, better ask @elpaso eg.
@tillhainbach fyi the doc repo is at https://github.com/qgis/QGIS-Documentation/ (and docs folder). You can also reach the file to edit directly from browsing the html doc and clicking the "Edit on github" button at the top of the page.

[asimrp]

I stumbled upon this exact issue and the workaround to use libsqlite3.dylib bundled with QGIS solved my problem as well. One question still remains unanswered - what is the root cause here? What exactly is different between MacOS provided libsqlite3 and the one bundled with QGIS? Are we adding custom modifications to libsqlite3, that are not available in the native version?

[PeterPetrik]

they have different version, they have different exported symbols and also differently mangled. We cannot use macOS native sqlite3 since they do not ship headers.

[asimrp]

Thank you for providing that information. Is this situation specific to MacOS or linux also has similar problems with libsqlite3?

[PeterPetrik]

linux is fine