Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

qgis_process dev+release+LTR abort on Ubuntu with ubuntugis-unstable PPA #56285

Closed
2 tasks
florisvdh opened this issue Feb 10, 2024 · 19 comments
Closed
2 tasks
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Build/Install Related to compiling or installing QGIS Linux Packaging Processing Relating to QGIS Processing framework or individual Processing algorithms

Comments

@florisvdh
Copy link
Contributor

florisvdh commented Feb 10, 2024

What is the bug or the crash?

--- UPDATE: currently the remaining problem is the one reproduced further below ---.

qgis_process from the master branch, either built locally in Linux Mint (Ubuntu Jammy), or installed from the ubuntugis-nightly repo, does not work, apart from returning the global help and its version info. It cannot load PyQGIS and it segfaults.

$ qgis_process plugins
./src/core/providers/qgsproviderregistry.cpp:367 : (init) [102ms] Loaded 33 providers (OAPIF;WFS;arcgisfeatureserver;arcgismapserver;arcgisvectortileservice;cesiumtiles;copc;delimitedtext;ept;gdal;gpx;grass;grassraster;hana;mbtilesvectortiles;mdal;memory;mesh_memory;mssql;ogr;pdal;postgres;postgresraster;spatialite;tiledscene;vectortile;virtual;virtualraster;vpc;vtpkvectortiles;wcs;wms;xyzvectortiles) 
./src/process/qgsprocess.cpp:209 : (loadPythonSupport) [141ms] load library /usr/lib/qgispython (3.35.0)
"<font color=\"red\">Couldn't load PyQGIS.<br>Python support will be disabled.</font><br><pre><br>Traceback (most recent call last):<br>&nbsp; File \"<string>\", line 1, in <module><br>&nbsp; File \"/usr/lib/python3/dist-packages/qgis/core/__init__.py\", line 25, in <module><br>&nbsp; &nbsp; from qgis._core import *<br>ModuleNotFoundError: No module named 'PyQt5.QtSerialPort'<br><br></pre>Python version:<br>3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]<br><br>QGIS version:<br>3.35.0-Master 'Master', 8e9267f5848<br><br>Python path:<br>['/usr/share/qgis/python', '/home/floris/.local/share/QGIS/QGIS3/profiles/default/python', '/home/floris/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/home/floris', '/usr/local/python', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.10/dist-packages']"
./src/core/qgsmessagelog.cpp:39 : (logMessage) [92ms] 2024-02-10T11:33:49 Python error[1] Couldn't load PyQGIS.
Python support will be disabled.

Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3/dist-packages/qgis/core/__init__.py", line 25, in 
    from qgis._core import *
ModuleNotFoundError: No module named 'PyQt5.QtSerialPort'

Python version:
3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

QGIS version:
3.35.0-Master 'Master', 8e9267f5848

Python path:
['/usr/share/qgis/python', '/home/floris/.local/share/QGIS/QGIS3/profiles/default/python', '/home/floris/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/home/floris', '/usr/local/python', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.10/dist-packages']
Segmentation fault (core dumped)
$ 
$ qgis_process help native:buffer
./src/core/providers/qgsproviderregistry.cpp:367 : (init) [93ms] Loaded 33 providers (OAPIF;WFS;arcgisfeatureserver;arcgismapserver;arcgisvectortileservice;cesiumtiles;copc;delimitedtext;ept;gdal;gpx;grass;grassraster;hana;mbtilesvectortiles;mdal;memory;mesh_memory;mssql;ogr;pdal;postgres;postgresraster;spatialite;tiledscene;vectortile;virtual;virtualraster;vpc;vtpkvectortiles;wcs;wms;xyzvectortiles) 
./src/process/qgsprocess.cpp:209 : (loadPythonSupport) [152ms] load library /usr/lib/qgispython (3.35.0)
"<font color=\"red\">Couldn't load PyQGIS.<br>Python support will be disabled.</font><br><pre><br>Traceback (most recent call last):<br>&nbsp; File \"<string>\", line 1, in <module><br>&nbsp; File \"/usr/lib/python3/dist-packages/qgis/core/__init__.py\", line 25, in <module><br>&nbsp; &nbsp; from qgis._core import *<br>ModuleNotFoundError: No module named 'PyQt5.QtSerialPort'<br><br></pre>Python version:<br>3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]<br><br>QGIS version:<br>3.35.0-Master 'Master', 8e9267f5848<br><br>Python path:<br>['/usr/share/qgis/python', '/home/floris/.local/share/QGIS/QGIS3/profiles/default/python', '/home/floris/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/home/floris', '/usr/local/python', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.10/dist-packages']"
./src/core/qgsmessagelog.cpp:39 : (logMessage) [91ms] 2024-02-10T11:34:51 Python error[1] Couldn't load PyQGIS.
Python support will be disabled.

Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3/dist-packages/qgis/core/__init__.py", line 25, in 
    from qgis._core import *
ModuleNotFoundError: No module named 'PyQt5.QtSerialPort'

Python version:
3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

QGIS version:
3.35.0-Master 'Master', 8e9267f5848

Python path:
['/usr/share/qgis/python', '/home/floris/.local/share/QGIS/QGIS3/profiles/default/python', '/home/floris/.local/share/QGIS/QGIS3/profiles/default/python/plugins', '/usr/share/qgis/python/plugins', '/home/floris', '/usr/local/python', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.10/dist-packages']
Segmentation fault (core dumped)

Steps to reproduce the issue

  1. Install QGIS from the ubuntugis-nightly repo
  2. In a shell, run qgis_process plugins or qgis_process help native:buffer.

Versions

$ qgis_process --version
QGIS 3.35.0-Master 'Master' (8e9267f5848)
QGIS code revision 8e9267f5848
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.8.3
PROJ version 9.3.1
EPSG Registry database version v10.098 (2023-11-24)
GEOS version 3.12.1-CAPI-1.18.1
SQLite version 3.37.2
OS Linux Mint 21.3
This copy of QGIS writes debugging output.
$ 
$ apt policy qgis
qgis:
  Installed: 1:3.35.0+git20240207+8e9267f5848+36jammy-ubuntugis
  Candidate: 1:3.35.0+git20240207+8e9267f5848+36jammy-ubuntugis
  Version table:
 *** 1:3.35.0+git20240207+8e9267f5848+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     1:3.35.0+git20240204+274c3da7543+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
     1:3.35.0+git20240201+d6008209dbc+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
     1:3.35.0+git20240128+4293d4f1c57+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
     1:3.35.0+git20240125+8c52728faf8+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
     3.28.15+dfsg-1~jammy1 500
        500 http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu jammy/main amd64 Packages
     3.22.4+dfsg-3build1 500
        500 http://ftp.belnet.be/ubuntu jammy/universe amd64 Packages

Supported QGIS version

  • I'm running a supported QGIS version according to the roadmap.

New profile

Additional context

The problem was observed already a few days ago with a locally built QGIS at 9631b3d; the make process already gives below errors. I presume it is the same build problem that applies to the QGIS builds for ubuntugis and maybe other ubuntu/Linux repositories.

An unexpected error occurred.
Check that you are using the latest version of PyQt5 and send an error report to
support@riverbankcomputing.com, including the following information:

  * your version of PyQt (5.15.6)
  * the UI file that caused this error
  * the debug output of pyuic5 (use the -d flag when calling pyuic5)
make[2]: *** [python/plugins/db_manager/CMakeFiles/zzz-db_manager-15-depend.dir/build.make:86: python/plugins/db_manager/ui_DlgAddGeometryColumn.py] Fout 1
make[2]: *** [python/console/CMakeFiles/pyconsole.dir/build.make:74: python/console/ui_console_compile_apis.py] Fout 1
make[2]: *** Verwijderen van bestand 'python/plugins/db_manager/ui_DlgAddGeometryColumn.py'
make[2]: *** Verwijderen van bestand 'python/console/ui_console_compile_apis.py'
make[2]: *** [python/plugins/db_manager/db_plugins/postgis/plugins/versioning/CMakeFiles/zzz-db_manager-5-depend.dir/build.make:75: python/plugins/db_manager/db_plugins/postgis/plugins/versioning/ui_DlgVersioning.py] Fout 1
make[2]: *** Verwijderen van bestand 'python/plugins/db_manager/db_plugins/postgis/plugins/versioning/ui_DlgVersioning.py'
make[1]: *** [CMakeFiles/Makefile2:7437: python/plugins/db_manager/CMakeFiles/zzz-db_manager-15-depend.dir/all] Fout 2
make[1]: *** Wachten op onvoltooide taken...
make[1]: *** [CMakeFiles/Makefile2:7739: python/plugins/db_manager/db_plugins/postgis/plugins/versioning/CMakeFiles/zzz-db_manager-5-depend.dir/all] Fout 2
make[1]: *** [CMakeFiles/Makefile2:9015: python/console/CMakeFiles/pyconsole.dir/all] Fout 2
[100%] Generating ../output/data/python/qsci_apis/PyQGIS.pap
[100%] Built target pyqgis-pap

I've attached my full local make output: make_errors.txt. I've had the same problem with at least 3 other newer versions since.

I've not timed these unsuccessful make processes, but in my perception they also take quite a bit lot longer (about 90 min instead of < 60 min ?), than about 10 days ago.

@florisvdh florisvdh added the Bug Either a bug report, or a bug fix. Let's hope for the latter! label Feb 10, 2024
@agiudiceandrea
Copy link
Contributor

agiudiceandrea commented Feb 10, 2024

@florisvdh, it looks like the PyQt5.QtSerialPort module is missing.
See #56256 d5b9669
I guess the issue with the QGIS version installed via ubuntugis-nightly repo will be fixed after a new build will be available.

@florisvdh
Copy link
Contributor Author

Indeed I could build without errors after installing the python3-pyqt5.qtserialport.

However qgis_process still aborts when finalizing:

$ ~/git_repositories2/QGIS/build-f68804f2/output/bin/qgis_process plugins
Available plugins
(* indicates loaded plugins which implement Processing providers)

  QNEAT3
  valhalla
  processing_saga_nextgen
  networks
  pcraster_tools
  ViewshedAnalysis
  cartography_tools
  network_gt
  wbt_for_qgis
  latlontools
  grassprovider
* processing
corrupted size vs. prev_size in fastbins
Aborted (core dumped)
$ 
$ ~/git_repositories2/QGIS/build-f68804f2/output/bin/qgis_process help native:buffer 1> /dev/null
corrupted size vs. prev_size in fastbins
Aborted (core dumped)
$ 
$ ~/git_repositories2/QGIS/build-f68804f2/output/bin/qgis_process --version
QGIS 3.35.0-Master 'Master' (f68804f2)
QGIS code revision f68804f2
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.8.3
PROJ version 9.3.1
EPSG Registry database version v10.098 (2023-11-24)
GEOS version 3.12.1-CAPI-1.18.1
SQLite version 3.37.2
OS Linux Mint 21.3

@agiudiceandrea agiudiceandrea added Build/Install Related to compiling or installing QGIS Processing Relating to QGIS Processing framework or individual Processing algorithms labels Feb 11, 2024
@nyalldawson
Copy link
Collaborator

@florisvdh did you try wiping your whole build directory and doing a full rebuild?

@florisvdh florisvdh changed the title qgis_process 'ubuntugis-nightly' (3.35) segfaults - "Couldn't load PyQGIS" qgis_process dev (3.35) aborts Feb 12, 2024
@florisvdh
Copy link
Contributor Author

I make each new build in a separate build subdirectory of the local git repo, e.g. build-f68804f2, etc. I assume that is indeed the 'full rebuild'?

$ ls -1 ~/git_repositories2/QGIS/build*/Makefile
/home/floris/git_repositories2/QGIS/build-d5b9669d/Makefile
/home/floris/git_repositories2/QGIS/build-f68804f2/Makefile

@agiudiceandrea agiudiceandrea added Feedback Waiting on the submitter for answers and removed Feedback Waiting on the submitter for answers labels Feb 12, 2024
@florisvdh
Copy link
Contributor Author

florisvdh commented Feb 14, 2024

I could still confirm this in e7f5b6f (2023-02-13).

@nyalldawson is this problem the explanation as to why no new builds are coming through in the ubuntugis-nightly repo? (https://ubuntugis.qgis.org/ubuntugis-nightly/pool/main/q/qgis/?C=M;O=D) The latest currently available build is 8e9267f from one week ago (2023-02-07); i.e. the release that led to this issue.

The latest ubuntu-nightly (which I didn't test) is from 2023-02-05, may be the same problem (https://ubuntu.qgis.org/ubuntu-nightly/pool/main/q/qgis/?C=M;O=D).

@florisvdh
Copy link
Contributor Author

Apparently the problem now also happens with QGIS from the ubuntugis-nightly-release repo. (corrupted size vs. prev_size in fastbins). (Seen in the qgisprocess R package repo GHA here)

@agiudiceandrea
Copy link
Contributor

Maybe @jef-n has some clues on what's happening.

@florisvdh
Copy link
Contributor Author

florisvdh commented Feb 15, 2024

Apparently the problem now also happens with QGIS from the ubuntugis-nightly-release repo. (corrupted size vs. prev_size in fastbins). (Seen in the qgisprocess R package repo GHA here)

Given that the QGIS revision used in the mentioned GHA workflow (ubuntugis-nightly-release 3.34.3 44e6e78 from 2023-02-03):

  • had already run successfully before, e.g. here on 2023-02-08,
  • is actually older than the moment where these problems started (the last succesful dev version (3.35) that I've built (or at least found trace of) was 8dc3558 from 2023-02-05)

... and given that the same problem has now also been observed by an R user on ubuntugis-LTR, which too provides a version that is still January (2023-01-21) – issue r-spatial/qgisprocess#205:

then it is probably because a change external to QGIS is going on in either Ubuntu or the ubuntugis-unstable PPA, that triggers the problem. I'm not sure whether it's the PPA:

  • arguments contra:
    • the successful ubuntugis-nightly-release GHA workflow (R package repo) on 2023-02-08 already used the more recent GDAL etc versions from the PPA. So that would mean it's not the PPA itself.
    • my latest successful QGIS dev build (8dc3558 from 2023-02-05) was already built against the ubuntugis-unstable PPA, which right then had just received its updates of GDAL etc.
  • argument pro: the latest GHA run (R package repo) only fails with the ubuntugis-nightly-release QGIS (3.34.3), not with ubuntu-nightly (3.35), ubuntu or ubuntu-ltr releases where the PPA is not used (note that the ubuntugis-nightly is not tested in the R pkg repo).

I have no clue at this moment. But I presume it will need a solution at the QGIS / qgis_process side.

@florisvdh florisvdh changed the title qgis_process dev (3.35) aborts qgis_process dev+release+LTR abort on Ubuntu with ubuntugis-unstable PPA Feb 15, 2024
@florisvdh
Copy link
Contributor Author

I've made an attempt at narrowing the problem further down. This is still a guess: could it be due to a PDAL update from the ubuntugis-unstable PPA that happened only on 9 Feb, which is several days after the other updates on this PPA? That would be consistent with all observations so far.

@nyalldawson could there be an incompatibility of qgis_process with PDAL 2.6.2 that was not there yet with PDAL 2.5.0 (my local builds use 'WITH_PDAL' set to ON, which is default)?

  • Note that this update also triggers the problem on existing builds (that were built against PDAL 2.5.0). So it also affects the published ubuntugis builds, including release and LTR.

On my system I see no other potential candidate package updates that would have triggered this problem:

$ grep -E "2024-02-(06|07|08|09|10|11).*(install |upgrade |remove )" /var/log/dpkg.log | grep -vE "r-cran|docker|qgis|qtserialport"
2024-02-08 11:24:19 upgrade firefox:amd64 122.0+linuxmint1+virginia 122.0.1+linuxmint1+virginia
2024-02-08 11:24:19 upgrade chromium:amd64 121.0.6167.139~linuxmint1+virginia 121.0.6167.160~linuxmint1+virginia
2024-02-08 11:24:25 upgrade firefox-locale-en:amd64 122.0+linuxmint1+virginia 122.0.1+linuxmint1+virginia
2024-02-08 11:24:25 upgrade firefox-locale-nl:amd64 122.0+linuxmint1+virginia 122.0.1+linuxmint1+virginia
2024-02-08 11:24:25 install linux-modules-5.15.0-94-generic:amd64 <none> 5.15.0-94.104
2024-02-08 11:24:26 install linux-image-5.15.0-94-generic:amd64 <none> 5.15.0-94.104
2024-02-08 11:24:26 install linux-modules-extra-5.15.0-94-generic:amd64 <none> 5.15.0-94.104
2024-02-08 11:24:27 upgrade linux-generic:amd64 5.15.0.92.89 5.15.0.94.91
2024-02-08 11:24:27 upgrade linux-image-generic:amd64 5.15.0.92.89 5.15.0.94.91
2024-02-08 11:24:27 install linux-headers-5.15.0-94:all <none> 5.15.0-94.104
2024-02-08 11:24:29 install linux-headers-5.15.0-94-generic:amd64 <none> 5.15.0-94.104
2024-02-08 11:24:30 upgrade linux-headers-generic:amd64 5.15.0.92.89 5.15.0.94.91
2024-02-08 11:24:30 upgrade linux-libc-dev:amd64 5.15.0-92.102 5.15.0-94.104
2024-02-09 09:43:51 upgrade libde265-dev:amd64 1.0.8-1 1.0.8-1ubuntu0.1
2024-02-09 09:43:52 upgrade libde265-0:amd64 1.0.8-1 1.0.8-1ubuntu0.1
2024-02-09 09:44:11 install libpdal16:amd64 <none> 2.6.2+ds-1~jammy0
2024-02-09 09:44:11 upgrade libpdal-dev:amd64 2.5.0+ds-1~jammy3 2.6.2+ds-1~jammy0
2024-02-09 09:44:11 upgrade libpdal-plugins:amd64 2.5.0+ds-1~jammy3 2.6.2+ds-1~jammy0
2024-02-09 09:44:11 upgrade libpdal-plugin-faux:amd64 2.5.0+ds-1~jammy3 2.6.2+ds-1~jammy0
2024-02-09 09:44:11 upgrade libpdal-plugin-hdf:amd64 2.5.0+ds-1~jammy3 2.6.2+ds-1~jammy0
2024-02-09 09:44:11 upgrade libpdal-plugin-icebridge:amd64 2.5.0+ds-1~jammy3 2.6.2+ds-1~jammy0
2024-02-09 09:44:11 upgrade libpdal-plugin-pgpointcloud:amd64 2.5.0+ds-1~jammy3 2.6.2+ds-1~jammy0
2024-02-09 09:44:11 upgrade libpdal-plugin-e57:amd64 2.5.0+ds-1~jammy3 2.6.2+ds-1~jammy0
2024-02-09 09:44:11 upgrade pdal:amd64 2.5.0+ds-1~jammy3 2.6.2+ds-1~jammy0
2024-02-09 19:24:07 upgrade cinnamon-control-center-dbg:amd64 6.0.0+virginia 6.0.1+virginia
2024-02-09 19:24:07 upgrade cinnamon-control-center:amd64 6.0.0+virginia 6.0.1+virginia
2024-02-09 19:24:07 upgrade cinnamon-control-center-data:all 6.0.0+virginia 6.0.1+virginia
2024-02-09 19:24:07 upgrade libcinnamon-control-center1:amd64 6.0.0+virginia 6.0.1+virginia
2024-02-09 19:24:07 upgrade cinnamon-session:amd64 6.0.3+virginia 6.0.4+virginia
2024-02-09 19:24:07 upgrade cinnamon-session-common:all 6.0.3+virginia 6.0.4+virginia
$
$ apt policy libpdal-dev
libpdal-dev:
  Installed: 2.6.2+ds-1~jammy0
  Candidate: 2.6.2+ds-1~jammy0
  Version table:
 *** 2.6.2+ds-1~jammy0 500
        500 http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     2.3.0+ds-2ubuntu4 500
        500 http://ftp.belnet.be/ubuntu jammy/universe amd64 Packages

FYI @temospena

@nyalldawson
Copy link
Collaborator

@florisvdh it's quite possible that the pdsl update is at fault, but my suspension is that this is a generic issue with the PPA build and not something specific to qgis_process. I gather it's also reproducible when closing the qgis app?

@florisvdh
Copy link
Contributor Author

florisvdh commented Feb 16, 2024

@nyalldawson, yes, this is reproducible by just using qgis_process on the command line (here).

The qgis app itself has no obvious problems when launching it.

@florisvdh
Copy link
Contributor Author

It's PDAL-related indeed!

I just built QGIS from current master (8261e0d), but now with the cmake flag WITH_PDAL set as OFF. (By default it's ON)

Now qgis_process behaves in a normal way again, i.e. not aborting.

$ ~/git_repositories2/QGIS/build-8261e0d2/output/bin/qgis_process plugins
Available plugins
(* indicates loaded plugins which implement Processing providers)

* QNEAT3
* valhalla
  processing_saga_nextgen
  networks
* pcraster_tools
  ViewshedAnalysis
* cartography_tools
* network_gt
* wbt_for_qgis
* latlontools
* grassprovider
* processing
$ 
$ ~/git_repositories2/QGIS/build-8261e0d2/output/bin/qgis_process help native:buffer 1> /dev/null
$ 
$ ~/git_repositories2/QGIS/build-8261e0d2/output/bin/qgis_process --version                      
QGIS 3.35.0-Master 'Master' (8261e0d2)
QGIS code revision 8261e0d2
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.8.3
PROJ version 9.3.1
EPSG Registry database version v10.098 (2023-11-24)
GEOS version 3.12.1-CAPI-1.18.1
SQLite version 3.37.2
OS Linux Mint 21.3

So there's something going wrong with the QGIS - PDAL interaction since PDAL upgraded from 2.5.0 to 2.6.2 in the ubuntugis-unstable PPA.

These are the PDAL packages installed on my system:

$ dpkg -l | grep pdal
ii  libpdal-base13                                2.5.0+ds-1~jammy3                                  amd64        Point Data Abstraction Library - libpdal_base
ii  libpdal-dev                                   2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - development files
ii  libpdal-plugin-draco                          2.5.0+ds-1~jammy3                                  amd64        Point Data Abstraction Library - Draco plugin
ii  libpdal-plugin-e57                            2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - E57 plugin
ii  libpdal-plugin-faux                           2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - Faux plugin
ii  libpdal-plugin-hdf                            2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - HDF plugin
ii  libpdal-plugin-icebridge                      2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - Icebridge plugin
ii  libpdal-plugin-pgpointcloud                   2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - PostgreSQL PointCloud plugin
ii  libpdal-plugins                               2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - plugins
ii  libpdal-util13                                2.5.0+ds-1~jammy3                                  amd64        Point Data Abstraction Library - libpdal_util
ii  libpdal16                                     2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - libpdal_base
ii  pdal                                          2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library

In the GHA 'ubuntugis' workflow of the qgisprocess R package, these are the PDAL-packages that are installed as a QGIS/GRASS/SAGA dependency:

libpdal-base13
libpdal-plugin-e57
libpdal-plugin-faux
libpdal-plugin-hdf
libpdal-plugin-icebridge
libpdal-plugin-pgpointcloud
libpdal-plugins
libpdal-util13
libpdal16

This is a little less packages than on my system, but in both cases qgis_process crashed (with PDAL enabled).

Some of these packages are still at 2.5.0; in both environments this includes libpdal-base13 and libpdal-util13. Both these are being installed from the ubuntugis-unstable PPA, which means they weren't updated to 2.6.2 back there like the other PDAL packages.

Could this be the problem, since QGIS relies on these:

$ apt depends libqgis-core3.34.3 | grep pdal
  Depends: libpdal-base13
  Depends: libpdal-util13

... while at the same time the PDAL packages that were updated to 2.6.2 rely on them too:

$ # REVERSE PDAL dependencies of libpdal-util13:
$ apt rdepends libpdal-util13 | grep -v qgis
libpdal-util13
Reverse Depends:
  libpdal-base13
  libpdal-plugin-i3s
  libpdal-plugin-draco
  libpdal-base13
  pdal
  libpdal-plugin-pgpointcloud
  libpdal-plugin-icebridge
  libpdal-plugin-i3s
  libpdal-plugin-hdf
  libpdal-plugin-e57
  libpdal-plugin-draco
  libpdal-dev

Should this be discussed further with the PPA maintainers? Or is it something to be fixed at the QGIS side nonetheless? @nyalldawson @jef-n

@florisvdh
Copy link
Contributor Author

florisvdh commented Feb 18, 2024

I'm still clueless whether the PPA mixing the two versions is a problem per se.

... while at the same time the PDAL packages that were updated to 2.6.2 rely on them too:

$ # REVERSE PDAL dependencies of libpdal-util13:
$ apt rdepends libpdal-util13 | grep -v qgis
libpdal-util13
Reverse Depends:
  libpdal-base13
  libpdal-plugin-i3s
  libpdal-plugin-draco
  libpdal-base13
  pdal
  libpdal-plugin-pgpointcloud
  libpdal-plugin-icebridge
  libpdal-plugin-i3s
  libpdal-plugin-hdf
  libpdal-plugin-e57
  libpdal-plugin-draco
  libpdal-dev

because those libpdal-plugin* reverse dependencies above only came in because of a recommended dependency by libpdal-base13 on libpdal-plugins:

$ apt depends libpdal-base13
libpdal-base13
  Depends: libc6
  Depends: libcurl4
  Depends: libgcc-s1
  Depends: libgdal34
  Depends: libgeotiff5
  Depends: libpdal-util13
  Depends: libssl3
  Depends: libstdc++6
  Depends: libxml2
  Depends: libzstd1
  Depends: zlib1g
  Recommends: libpdal-plugins

The qgis packages only depend on libpdal-base13 and libpdal-util13.

Indeed, omitting their recommended and suggested dependencies show no sign of other PDAL packages.

$ apt depends --no-recommends --no-suggests libpdal-base13
libpdal-base13
  Depends: libc6
  Depends: libcurl4
  Depends: libgcc-s1
  Depends: libgdal34
  Depends: libgeotiff5
  Depends: libpdal-util13
  Depends: libssl3
  Depends: libstdc++6
  Depends: libxml2
  Depends: libzstd1
  Depends: zlib1g
$ apt depends --no-recommends --no-suggests libpdal-util13
libpdal-util13
  Depends: libc6
  Depends: libgcc-s1
  Depends: libstdc++6

Still, it's clear that the PPA-update of the libpdal-plugin* packages from 2.5.0 to 2.6.2, and/or the newly installed package pdal16 (2.6.2) did cause the QGIS-PDAL issue to emerge.

Below is the visualisation by debtree for some of these dependencies. The blue lines also show version requirements.

Legend:

       Depends:                 blue
       Recommends:              black

out2_clip

The complete visual can be obtained by:

$ debtree --max-depth=3 libpdal-base13 > out.dot
$ dot -T png -o out.png out.dot

@florisvdh
Copy link
Contributor Author

florisvdh commented Feb 18, 2024

It can also be shown that libpdal16 2.6.2 and its reverse dependencies are at stake by the following workaround.

Removing libpdal16 lets qgis_process from the QGIS ubuntugis repository (release 3.34.3) return to normal behaviour:

$ # bad behaviour
$ qgis_process help native:buffer 1> /dev/null
corrupted size vs. prev_size in fastbins
Aborted (core dumped)
$ 
$ apt purge libpdal16        
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  libpdal-plugin-e57* libpdal-plugin-faux* libpdal-plugin-hdf* libpdal-plugin-icebridge* libpdal-plugin-pgpointcloud* libpdal-plugins* libpdal16*
0 upgraded, 0 newly installed, 7 to remove and 10 not upgraded.
After this operation, 8441 kB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 682796 files and directories currently installed.)
Removing libpdal-plugins (2.6.2+ds-1~jammy0) ...
Removing libpdal-plugin-e57 (2.6.2+ds-1~jammy0) ...
Removing libpdal-plugin-faux (2.6.2+ds-1~jammy0) ...
Removing libpdal-plugin-hdf (2.6.2+ds-1~jammy0) ...
Removing libpdal-plugin-icebridge (2.6.2+ds-1~jammy0) ...
Removing libpdal-plugin-pgpointcloud (2.6.2+ds-1~jammy0) ...
Removing libpdal16 (2.6.2+ds-1~jammy0) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
$ 
$ # good behaviour
$ qgis_process help native:buffer 1> /dev/null
$ 
$ qgis_process plugins 1> /dev/null
$ 
$ qgis_process --version
QGIS 3.34.3-Prizren 'Prizren' (47373234acd)
QGIS code revision 47373234acd
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.6.4
Compiled against PROJ 9.1.1
Running against PROJ 9.3.1
EPSG Registry database version v10.098 (2023-11-24)
Compiled against GEOS 3.11.1-CAPI-1.17.1
Running against GEOS 3.12.1-CAPI-1.18.1
SQLite version 3.37.2
OS Linux Mint 21.3
$ 
$ apt policy qgis
qgis:
  Installed: 1:3.34.3+36jammy-ubuntugis
  Candidate: 1:3.34.3+36jammy-ubuntugis
  Version table:
 *** 1:3.34.3+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     1:3.34.2+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis jammy/main amd64 Packages
     1:3.34.1+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis jammy/main amd64 Packages
     3.28.15+dfsg-1~jammy1 500
        500 https://ppa.launchpadcontent.net/ubuntugis/ubuntugis-unstable/ubuntu jammy/main amd64 Packages
     3.22.4+dfsg-3build1 500
        500 http://ftp.belnet.be/ubuntu jammy/universe amd64 Packages
$ 
$ dpkg -l | grep pdal
ii  libpdal-base13                                2.5.0+ds-1~jammy3                                 amd64        Point Data Abstraction Library - libpdal_base
ii  libpdal-plugin-draco                          2.5.0+ds-1~jammy3                                 amd64        Point Data Abstraction Library - Draco plugin
ii  libpdal-util13                                2.5.0+ds-1~jammy3                                 amd64        Point Data Abstraction Library - libpdal_util

But perhaps this would render a series of PDAL algorithms non-functional, not sure about that.

$ qgis_process list | grep -i pdal
QGIS (PDAL)
	pdal:assignprojection	Assign projection
	pdal:boundary	Boundary
	pdal:clip	Clip
	pdal:convertformat	Convert format
	pdal:createcopc	Create COPC
	pdal:density	Density
	pdal:exportraster	Export to raster
	pdal:exportrastertin	Export to raster (using triangulation)
	pdal:exportvector	Export to vector
	pdal:filter	Filter
	pdal:info	Information
	pdal:merge	Merge
	pdal:reproject	Reproject
	pdal:thinbydecimate	Thin (by skipping points)
	pdal:thinbyradius	Thin (by sampling radius)
	pdal:tile	Tile
	pdal:virtualpointcloud	Build virtual point cloud (VPC)

@florisvdh
Copy link
Contributor Author

florisvdh commented Feb 23, 2024

Current behaviour of QGIS 3.35.0 in ubuntugis-nightly has changed. QGIS now no longer depends on libpdal-util13, but it needs libpdal16.

My observation is that it works perfect for qgis_process on condition that libpdal-util13 and its reverse dependencies have been removed.

@jef-n I think the libpdal-util13 2.5.0 and reverse dependencies should be automatically removed upon this upgrade (to be defined as a conflict?). Otherwise qgis_process keeps complaining.

Installation information:

Note that libpdal16 and libpdal-plugins were automatically re-installed as a consequence of the latest ubuntugis-nightly upgrade; it is now a regular dependency.

$ dpkg -l | grep libpdal16        
ii  libpdal16                                     2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - libpdal_base
$ 
$ dpkg -l | grep libpdal-plugin 
ii  libpdal-plugin-e57                            2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - E57 plugin
ii  libpdal-plugin-faux                           2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - Faux plugin
ii  libpdal-plugin-hdf                            2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - HDF plugin
ii  libpdal-plugin-icebridge                      2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - Icebridge plugin
ii  libpdal-plugin-pgpointcloud                   2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - PostgreSQL PointCloud plugin
ii  libpdal-plugins                               2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - plugins
$ 
$ apt depends libqgis-core3.35.0 | grep pdal
  Depends: libpdal16
$ 
$ qgis_process --version
QGIS 3.35.0-Master 'Master' (f6e073f0eed)
QGIS code revision f6e073f0eed
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.8.3
PROJ version 9.3.1
EPSG Registry database version v10.098 (2023-11-24)
GEOS version 3.12.1-CAPI-1.18.1
SQLite version 3.37.2
OS Linux Mint 21.3
This copy of QGIS writes debugging output.
$ 
$ apt policy qgis
qgis:
  Installed: 1:3.35.0+git20240221+f6e073f0eed+36jammy-ubuntugis
  Candidate: 1:3.35.0+git20240221+f6e073f0eed+36jammy-ubuntugis
  Version table:
 *** 1:3.35.0+git20240221+f6e073f0eed+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
        100 /var/lib/dpkg/status
     1:3.35.0+git20240218+b572110ec4f+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
     1:3.35.0+git20240213+02d2dd4ccc7+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
     1:3.35.0+git20240207+8e9267f5848+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
     1:3.35.0+git20240204+274c3da7543+36jammy-ubuntugis 500
        500 https://qgis.org/ubuntugis-nightly jammy/main amd64 Packages
     3.28.15+dfsg-1~jammy1 500
        500 http://ppa.launchpad.net/ubuntugis/ubuntugis-unstable/ubuntu jammy/main amd64 Packages
     3.22.4+dfsg-3build1 500
        500 http://ftp.belnet.be/ubuntu jammy/universe amd64 Packages

Before removing libpdal-util13: still aborting

$ qgis_process plugins 1> /dev/null
./src/core/providers/qgsproviderregistry.cpp:372 : (init) [91ms] Loaded 34 providers (OAPIF;WFS;arcgisfeatureserver;arcgismapserver;arcgisvectortileservice;cesiumtiles;copc;delimitedtext;ept;gdal;gpx;grass;grassraster;hana;mbtilesvectortiles;mdal;memory;mesh_memory;mssql;ogr;pdal;postgres;postgresraster;sensorthings;spatialite;tiledscene;vectortile;virtual;virtualraster;vpc;vtpkvectortiles;wcs;wms;xyzvectortiles) 
./src/process/qgsprocess.cpp:210 : (loadPythonSupport) [118ms] load library /usr/lib/qgispython (3.35.0)
double free or corruption (!prev)
Aborted (core dumped)
$ 
$ qgis_process help native:buffer 1> /dev/null
./src/core/providers/qgsproviderregistry.cpp:372 : (init) [89ms] Loaded 34 providers (OAPIF;WFS;arcgisfeatureserver;arcgismapserver;arcgisvectortileservice;cesiumtiles;copc;delimitedtext;ept;gdal;gpx;grass;grassraster;hana;mbtilesvectortiles;mdal;memory;mesh_memory;mssql;ogr;pdal;postgres;postgresraster;sensorthings;spatialite;tiledscene;vectortile;virtual;virtualraster;vpc;vtpkvectortiles;wcs;wms;xyzvectortiles) 
./src/process/qgsprocess.cpp:210 : (loadPythonSupport) [118ms] load library /usr/lib/qgispython (3.35.0)
double free or corruption (!prev)
Aborted (core dumped)

Removing libpdal-util13:

$ apt purge libpdal-util13 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libgdal32 liblaszip-api8 liblaszip-dev liblaszip8 linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic linux-image-5.15.0-92-generic linux-modules-5.15.0-92-generic
  linux-modules-extra-5.15.0-92-generic
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  libpdal-base13* libpdal-plugin-draco* libpdal-util13*
0 upgraded, 0 newly installed, 3 to remove and 4 not upgraded.
After this operation, 6410 kB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 607439 files and directories currently installed.)
Removing libpdal-plugin-draco (2.5.0+ds-1~jammy3) ...
Removing libpdal-base13 (2.5.0+ds-1~jammy3) ...
Removing libpdal-util13 (2.5.0+ds-1~jammy3) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...

After removing libpdal-util13: success!

$ qgis_process plugins 1> /dev/null
./src/core/providers/qgsproviderregistry.cpp:372 : (init) [81ms] Loaded 34 providers (OAPIF;WFS;arcgisfeatureserver;arcgismapserver;arcgisvectortileservice;cesiumtiles;copc;delimitedtext;ept;gdal;gpx;grass;grassraster;hana;mbtilesvectortiles;mdal;memory;mesh_memory;mssql;ogr;pdal;postgres;postgresraster;sensorthings;spatialite;tiledscene;vectortile;virtual;virtualraster;vpc;vtpkvectortiles;wcs;wms;xyzvectortiles) 
./src/process/qgsprocess.cpp:210 : (loadPythonSupport) [122ms] load library /usr/lib/qgispython (3.35.0)
$ 
$ qgis_process help native:buffer 1> /dev/null
./src/core/providers/qgsproviderregistry.cpp:372 : (init) [82ms] Loaded 34 providers (OAPIF;WFS;arcgisfeatureserver;arcgismapserver;arcgisvectortileservice;cesiumtiles;copc;delimitedtext;ept;gdal;gpx;grass;grassraster;hana;mbtilesvectortiles;mdal;memory;mesh_memory;mssql;ogr;pdal;postgres;postgresraster;sensorthings;spatialite;tiledscene;vectortile;virtual;virtualraster;vpc;vtpkvectortiles;wcs;wms;xyzvectortiles) 
./src/process/qgsprocess.cpp:210 : (loadPythonSupport) [118ms] load library /usr/lib/qgispython (3.35.0)

@jef-n
Copy link
Member

jef-n commented Feb 23, 2024

Actually removing libpdal-plugin-draco alone is enough, which IMHO makes it a pure PDAL issue. PDAL 2.6 probably fails to load (or unload) it's old plugin and crashes. No sure why there isn't a current draco plugin. I guess libpdal16 should just conflict with libpdal-base13. @kalxas any insights?

@florisvdh
Copy link
Contributor Author

You're right @jef-n ! Thanks.

$ dpkg -l | grep -E "pdal16|pdal.*2\.5"
ii  libpdal-base13                                2.5.0+ds-1~jammy3                                  amd64        Point Data Abstraction Library - libpdal_base
ii  libpdal-plugin-draco                          2.5.0+ds-1~jammy3                                  amd64        Point Data Abstraction Library - Draco plugin
ii  libpdal-util13                                2.5.0+ds-1~jammy3                                  amd64        Point Data Abstraction Library - libpdal_util
ii  libpdal16                                     2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - libpdal_base
$ 
$ qgis_process plugins 1> /dev/null
./src/core/providers/qgsproviderregistry.cpp:372 : (init) [95ms] Loaded 34 providers (OAPIF;WFS;arcgisfeatureserver;arcgismapserver;arcgisvectortileservice;cesiumtiles;copc;delimitedtext;ept;gdal;gpx;grass;grassraster;hana;mbtilesvectortiles;mdal;memory;mesh_memory;mssql;ogr;pdal;postgres;postgresraster;sensorthings;spatialite;tiledscene;vectortile;virtual;virtualraster;vpc;vtpkvectortiles;wcs;wms;xyzvectortiles) 
./src/process/qgsprocess.cpp:210 : (loadPythonSupport) [124ms] load library /usr/lib/qgispython (3.35.0)
double free or corruption (!prev)
Aborted (core dumped)
$ 
$ apt purge libpdal-plugin-draco
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libgdal32 liblaszip-api8 liblaszip-dev liblaszip8 libpdal-base13 libpdal-util13 linux-headers-5.15.0-92 linux-headers-5.15.0-92-generic linux-image-5.15.0-92-generic
  linux-modules-5.15.0-92-generic linux-modules-extra-5.15.0-92-generic
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  libpdal-plugin-draco*
0 upgraded, 0 newly installed, 1 to remove and 4 not upgraded.
After this operation, 401 kB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 607439 files and directories currently installed.)
Removing libpdal-plugin-draco (2.5.0+ds-1~jammy3) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...
$ 
$ qgis_process plugins 1> /dev/null
./src/core/providers/qgsproviderregistry.cpp:372 : (init) [95ms] Loaded 34 providers (OAPIF;WFS;arcgisfeatureserver;arcgismapserver;arcgisvectortileservice;cesiumtiles;copc;delimitedtext;ept;gdal;gpx;grass;grassraster;hana;mbtilesvectortiles;mdal;memory;mesh_memory;mssql;ogr;pdal;postgres;postgresraster;sensorthings;spatialite;tiledscene;vectortile;virtual;virtualraster;vpc;vtpkvectortiles;wcs;wms;xyzvectortiles) 
./src/process/qgsprocess.cpp:210 : (loadPythonSupport) [119ms] load library /usr/lib/qgispython (3.35.0)
$ 
$ dpkg -l | grep -E "pdal16|pdal.*2\.5"
ii  libpdal-base13                                2.5.0+ds-1~jammy3                                  amd64        Point Data Abstraction Library - libpdal_base
ii  libpdal-util13                                2.5.0+ds-1~jammy3                                  amd64        Point Data Abstraction Library - libpdal_util
ii  libpdal16                                     2.6.2+ds-1~jammy0                                  amd64        Point Data Abstraction Library - libpdal_base

@florisvdh
Copy link
Contributor Author

From apt autoremove one can see that libpdal-base13 and libpdal-util13 have become orphans after removing the draco plugin alone. Removing libpdal-util13 triggers the removal of all three 2.5.0 packages (ie including the draco plugin).

But indeed it seems it's the draco plugin that generates a problem when it's present together with libpdal16.

@jef-n
Copy link
Member

jef-n commented Feb 24, 2024

Filed https://trac.osgeo.org/ubuntugis/ticket/87

@jef-n jef-n closed this as completed Feb 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Either a bug report, or a bug fix. Let's hope for the latter! Build/Install Related to compiling or installing QGIS Linux Packaging Processing Relating to QGIS Processing framework or individual Processing algorithms
Projects
None yet
Development

No branches or pull requests

4 participants