Cannot load qgis_process - corrupted size vs. prev_size in fastbins Aborted (core dumped)

[temospena]

Things to check beforehand

• To avoid duplication, I have checked existing issues.
• I use a currently supported QGIS version, i.e. shown green in the QGIS release schedule.

Are QGIS plugins the cause?

The problem remains with all plugins disabled.

Description of the observed problem

I cannot load qgis_process in R - apparently it does not find qgisprocess.
But qgis_process exists, and lives in the default place (see minimal reproducible example).

In terminal, when calling qgis_process or any other variaton (qgis_process --list, qgis_process --version, qgis_process plugins), it shows some stuff but always ends with

double free or corruption (!prev)
Aborted (core dumped)

(see entire output bellow)

I would say that both are related (this problem is affecting the correct loading in R).
I am struggling with this because in another machine I have the exactly same setup and it runs smoothly.

I also tried to free memory with free -h && sudo sysctl -w vm.drop_caches=3 && sudo sync && echo 3 | sudo tee /proc/sys/vm/drop_caches && free -h but it does not solve anything.

I purged and reinstalled qgis and r-spatial/qgisprocess, and the problem remains.

Any suggestion would be helpful :)

Minimal reproducible example

> remotes::install_github("r-spatial/qgisprocess")
Using github PAT from envvar GITHUB_TOKEN
Skipping install of 'qgisprocess' from a github remote, the SHA1 (2d02f89d) has not changed since last install.
  Use `force = TRUE` to force installation
> library(qgisprocess)
> qgis_path()
NULL
> options(qgisprocess.path = "usr/bin/qgis_process")
> library("qgisprocess")
> qgis_configure()
Trying getOption('qgisprocess.path'): 'usr/bin/qgis_process'
Running 'usr/bin/qgis_process' was not successful. Please check the qgisprocess.path option.
Error message was:

Error in c("process_initialize(self, private, command, args, stdin, stdout, ", : ! Native call to `processx_exec` failed
Caused by error in `chain_call(c_processx_exec, command, c(command, args), pty, pty_options, …`:
! cannot start processx process 'usr/bin/qgis_process' (system error 2, No such file or directory) @unix/processx.c:611 (processx_exec)

Sys.getenv('R_QGISPROCESS_PATH') was not found.
Trying 'qgis_process' on PATH...
'qgis_process' is not available on PATH.
No QGIS installation containing 'qgis_process' found!

in terminal

rosa@rosa-ist:~$ qgis_process
QGIS Processing Executor - 3.28.15-Firenze 'Firenze' (3.28.15-Firenze)
Usage: /usr/bin/qgis_process.bin [--help] [--version] [--json] [--verbose] [--no-python] [command] [algorithm id, path to model file, or path to Python script] [parameters]

Options:
        --help or -h            Output the help
        --version or -v         Output all versions related to QGIS Process
        --json          Output results as JSON objects
        --verbose       Output verbose logs
        --no-python     Disable Python support (results in faster startup)
Available commands:
        plugins         list available and active plugins
        plugins enable  enables an installed plugin. The plugin name must be specified, e.g. "plugins enable cartography_tools"
        plugins disable disables an installed plugin. The plugin name must be specified, e.g. "plugins disable cartography_tools"
        list            list all available processing algorithms
        help            show help for an algorithm. The algorithm id or a path to a model file must be specified.
        run             runs an algorithm. The algorithm id or a path to a model file and parameter values must be specified. Parameter values are specified after -- with PARAMETER=VALUE syntax. Ordered list values for a parameter can be created by specifying the parameter multiple times, e.g. --LAYERS=layer1.shp --LAYERS=layer2.shp
                        Alternatively, a '-' character in place of the parameters argument indicates that the parameters should be read from STDIN as a JSON object. The JSON should be structured as a map containing at least the "inputs" key specifying a map of input parameter values. This implies the --json option for output as a JSON object.
                        If required, the ellipsoid to use for distance and area calculations can be specified via the "--ELLIPSOID=name" argument.
                        If required, an existing QGIS project to use during the algorithm execution can be specified via the "--PROJECT_PATH=path" argument.
                        When passing parameters as a JSON object from STDIN, these extra arguments can be provided as an "ellipsoid" and a "project_path" key respectively.
corrupted size vs. prev_size in fastbins
Aborted (core dumped)

rosa@rosa-ist:~$ qgis_process --version
QGIS 3.28.15-Firenze 'Firenze' (252ad49ddcb)
QGIS code revision 252ad49ddcb
Qt version 5.15.3
Python version 3.10.12
GDAL/OGR version 3.6.4
Compiled against PROJ 9.1.1
Running against PROJ 9.3.1
EPSG Registry database version v10.098 (2023-11-24)
Compiled against GEOS 3.11.1-CAPI-1.17.1
Running against GEOS 3.12.1-CAPI-1.18.1
SQLite version 3.37.2
OS Ubuntu 22.04.3 LTS
double free or corruption (!prev)
Aborted (core dumped)

Backend versions

> qgis_version(query = TRUE, debug = TRUE)
The filepath of 'qgis_process' is not present in the package cache, so the package is not well configured.
Restart R and reload the package; run `qgis_configure()` if needed.
For now, will try to fix it on the fly, but some functionality may not work.

Trying getOption('qgisprocess.path'): 'usr/bin/qgis_process'
Running 'usr/bin/qgis_process' was not successful. Please check the qgisprocess.path option.
Error message was:

Error in c("process_initialize(self, private, command, args, stdin, stdout, ", : ! Native call to `processx_exec` failed
Caused by error in `chain_call(c_processx_exec, command, c(command, args), pty, pty_options, …`:
! cannot start processx process 'usr/bin/qgis_process' (system error 2, No such file or directory) @unix/processx.c:611 (processx_exec)

Sys.getenv('R_QGISPROCESS_PATH') was not found.
Trying 'qgis_process' on PATH...
'qgis_process' is not available on PATH.
The filepath of 'qgis_process' is not present in the package cache, so the package is not well configured.
Restart R and reload the package; run `qgis_configure()` if needed.
For now, will try to fix it on the fly, but some functionality may not work.

Trying getOption('qgisprocess.path'): 'usr/bin/qgis_process'
Running 'usr/bin/qgis_process' was not successful. Please check the qgisprocess.path option.
Error message was:

Error in c("process_initialize(self, private, command, args, stdin, stdout, ", : ! Native call to `processx_exec` failed
Caused by error in `chain_call(c_processx_exec, command, c(command, args), pty, pty_options, …`:
! cannot start processx process 'usr/bin/qgis_process' (system error 2, No such file or directory) @unix/processx.c:611 (processx_exec)

Sys.getenv('R_QGISPROCESS_PATH') was not found.
Trying 'qgis_process' on PATH...
'qgis_process' is not available on PATH.
Error in `qgis_query_path()`:
! No QGIS installation containing 'qgis_process' found!
Run `rlang::last_trace()` to see where the error occurred.

QGIS plugin state

> qgis_plugins(query = TRUE)
Error in if (qgis_using_json_output()) { : argument is of length zero

But they exist

rosa@rosa-ist:~$ qgis_process plugins
Problem with SAGA installation: unsupported SAGA version (found: 9.3.0, required: 2.3.).
Available plugins
(* indicates loaded plugins which implement Processing providers)

* processing_r
  otbprovider
* grassprovider
* processing
* sagaprovider
double free or corruption (!prev)
Aborted (core dumped)

R session info

> sessioninfo::session_info()
─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 setting  value
 version  R version 4.3.2 (2023-10-31)
 os       Ubuntu 22.04.3 LTS
 system   x86_64, linux-gnu
 ui       RStudio
 language (EN)
 collate  en_US.UTF-8
 ctype    pt_PT.UTF-8
 tz       Europe/Lisbon
 date     2024-02-15
 rstudio  2023.12.1+402 Ocean Storm (desktop)
 pandoc   2.9.2.1 @ /usr/bin/pandoc

─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 package     * version    date (UTC) lib source
 assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.3.1)
 cli           3.6.2      2023-12-11 [2] CRAN (R 4.3.2)
 glue          1.7.0      2024-01-09 [2] CRAN (R 4.3.2)
 processx      3.8.3      2023-12-10 [3] CRAN (R 4.3.2)
 ps            1.7.6      2024-01-18 [3] CRAN (R 4.3.2)
 qgisprocess * 0.3.0.9000 2024-02-15 [1] Github (r-spatial/qgisprocess@2d02f89)
 R6            2.5.1      2021-08-19 [3] CRAN (R 4.2.0)
 rappdirs      0.3.3      2021-01-31 [3] CRAN (R 4.2.0)
 remotes       2.4.2.1    2023-07-18 [1] CRAN (R 4.3.1)
 rJava       * 1.0-10     2023-12-02 [2] CRAN (R 4.3.2)
 rlang         1.1.3      2024-01-10 [2] CRAN (R 4.3.2)
 rstudioapi    0.15.0     2023-07-07 [3] CRAN (R 4.3.1)
 sessioninfo   1.2.2      2021-12-06 [3] CRAN (R 4.2.0)
 withr         3.0.0      2024-01-16 [2] CRAN (R 4.3.2)

 [1] /home/rosa/R/x86_64-pc-linux-gnu-library/4.3
 [2] /usr/local/lib/R/site-library
 [3] /usr/lib/R/site-library
 [4] /usr/lib/R/library

Additional context

I have QGIS installed following the official instructions.
My /etc/apt/sources.list.d/qgis.sources is:

Types: deb deb-src
URIs: https://qgis.org/ubuntugis-ltr
Suites: jammy
Architectures: amd64
Components: main
Signed-By: /etc/apt/keyrings/qgis-archive-keyring.gpg

I installed r-spatial/qgisprocess with
Rscript -e "remotes::install_github('r-spatial/qgisprocess')"

[florisvdh]

Thanks @temospena for your elaborate analysis, it may add more hints to solve this problem in QGIS. The problem pops up with several QGIS-versions built to work with library versions from the ubuntugis-unstable PPA, like your LTR version (which I didn't test, so it's good to know).

It's definitely the qgis_process crash which you demonstrate, that causes the problems in R.

The problem has been reported here: qgis/QGIS#56285. Initially there was yet another problem but this one still remains. The problem also just started popping up in the GitHub Actions package check here for the ubuntugis QGIS build.

It's actually the first time that I also see the crash happening with a mere qgis_process --version call; until now I only had it with a few other commands.

because in another machine I have the exactly same setup and it runs smoothly

This is interesting too. There should be some difference that explains it. Did you already update the system packages on that other system? Is the output of the qgis_process --version command exactly the same, and the PPA is also enabled?

[temospena]

Thank you @florisvdh for the analysis.

Regarding the other machine, something weird is happening.
It loads easily in R

>library(qgisprocess)
Attempting to load the package cache ... Success!
QGIS version: 3.28.15-Firenze
Having access to 653 algorithms from 5 QGIS processing providers.
Run `qgis_configure(use_cached_data = TRUE)` to reload cache and get more details.

but in terminal this message is thrown (and same for qgis_process --version):

~$ qgis_process
qt.qpa.xcb: could not connect to display 
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, xcb.

Aborted (core dumped)

But it works fine within R!

Regarding the ppa, i have these:

/etc/apt/sources.list.d/ubuntugis-ubuntu-ubuntugis-unstable-jammy.list:deb https://ppa.launchpadcontent.net/ubuntugis/ubuntugis-unstable/ubuntu/ jammy main
/etc/apt/sources.list.d/ubuntugis-ubuntu-ubuntugis-unstable-jammy.list:# deb-src https://ppa.launchpadcontent.net/ubuntugis/ubuntugis-unstable/ubuntu/ jammy main
/etc/apt/sources.list.d/ubuntugis-ubuntu-ubuntugis-unstable-jammy.list.save:deb https://ppa.launchpadcontent.net/ubuntugis/ubuntugis-unstable/ubuntu/ jammy main
/etc/apt/sources.list.d/ubuntugis-ubuntu-ubuntugis-unstable-jammy.list.save:# deb-src https://ppa.launchpadcontent.net/ubuntugis/ubuntugis-unstable/ubuntu/ jammy main

And I can see the differences between the 2 machines, when I do apt update & apt upgrade to the one where qgis_process is working:

The following NEW packages will be installed:
  linux-headers-6.5.0-18-generic linux-hwe-6.5-headers-6.5.0-18 linux-image-6.5.0-18-generic
  linux-modules-6.5.0-18-generic linux-modules-extra-6.5.0-18-generic linux-modules-nvidia-535-6.5.0-18-generic
  linux-objects-nvidia-535-6.5.0-18-generic linux-signatures-nvidia-6.5.0-18-generic
The following packages will be upgraded:
  base-files coreutils dns-root-data gdm3 gh gir1.2-gdm-1.0 iptables language-pack-en language-pack-en-base
  libegl-mesa0 libgbm1 libgdm1 libgl1-mesa-dev libgl1-mesa-dri libgl1-mesa-dri:i386 libglapi-mesa
  libglapi-mesa:i386 libglx-mesa0 libglx-mesa0:i386 libip4tc2 libip6tc2 libxatracker2 libxtables12
  linux-generic-hwe-22.04 linux-headers-generic-hwe-22.04 linux-image-generic-hwe-22.04
  linux-modules-nvidia-535-generic-hwe-22.04 login mesa-common-dev mesa-va-drivers mesa-vdpau-drivers
  mesa-vulkan-drivers passwd python-apt-common python3-apt python3-distupgrade tailscale
  ubuntu-release-upgrader-core ubuntu-release-upgrader-gtk uidmap

I did not proceed with the installation because qgis_process is working there and I need to use it :)

Still, for further investigation, I get the following in the both machines:

$  apt policy libpdal-dev
libpdal-dev:
  Installed: (none)
  Candidate: 2.6.2+ds-1~jammy0
  Version table:
     2.6.2+ds-1~jammy0 500
        500 https://ppa.launchpadcontent.net/ubuntugis/ubuntugis-unstable/ubuntu jammy/main amd64 Packages
     2.3.0+ds-2ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

Is there anything else that I should look at?

Anyway, maybe I'll try to install the LTR stable version (https://qgis.org/ubuntu-ltr) and check if it works. 🤔

[temospena]

Anyway, maybe I'll try to install the LTR stable version (https://qgis.org/ubuntu-ltr) and check if it works. 🤔

Yes, it works fine with the qgis LTR stable version.

I will wait for an update that solves this issue before going back to unstable.

[florisvdh]

Thanks, interesting!

OK, good to know that LTR stable works fine for you (so far). But it is not built against more recent GDAL, GEOS etc, which you already have installed on your system (from the ubuntugis-unstable PPA = more recent versions than those from Ubuntu repo; but the ubuntu-ltr QGIS is built against the latter). So one could get in trouble when doing processing since that QGIS was built against older versions. The version differences should be explicit from qgis_process --version.

The problems with running qgis_process --version on your other machine might be solved be running:

QT_QPA_PLATFORM=offscreen qgis_process --version

That is how all qgis_process calls are run by the R package, and would explain why you didn't have a problem from R.

Oh yes, the libpdal-dev library is probably only needed to build QGIS locally.

You could just run dpkg -l | grep pdal to see which PDAL-related packages you have, and at which version they currently are. Maybe there's a difference between both machines.

[florisvdh]

My best current workaround so far is the following. I expect it to work for all QGIS 'ubuntugis' releases (meant to work with ubuntugis-unstable PPA = including the GDAL, GEOS, PROJ versions distributed from there and which you have installed):

$ apt purge libpdal16

It will also remove a series of PDAL plugins (listed below).

libpdal-plugin-e57 libpdal-plugin-faux libpdal-plugin-hdf libpdal-plugin-icebridge libpdal-plugin-pgpointcloud libpdal-plugins

This appears to make the qgis_process backend work fine again.

I'm not sure about potential consequences though, in case you're using PDAL algorithms. Any further observations of potential side effects are welcome.

Of course this is not an ideal solution, but it will probably help in the meantime. I wait for further follow-up at qgis/QGIS#56285.

Reverting the removal of all above packages, after the QGIS-PDAL issue is resolved, can be done with a simple apt install libpdal16, at least with the current state of the PPA.

[jannes-m]

Hey @florisvdh,
I also ran into the double free or corruption (!prev) Aborted issue. Purging libpdal16 solved the problem, thanks for sharing!

[temospena]

I confirm.
Just upgraded to ubuntugis-ltr (changing the qgis.sources)
update & upgrade
qgis_process --version returned the error
apt purge libpdal16
qgis_process --version looks fine

I could run the processes from R smoothly 👌

[florisvdh]

Based on current observations with latest QGIS 3.35.0 for ubuntugis (ubuntugis-nightly repo of QGIS), the new QGIS ubuntugis releases (and at least upcoming 3.36) will automatically (re)install libpdal16 and libpdal-plugins, since they're now obligate dependencies.

This (re)introduces the aborting problem.

However libpdal-util13 (and its reverse dependencies) stopped being a dependency in these releases. If needed, actively remove it (apt purge libpdal-util13) and the problem is solved again. That state is of course what we need, going forward. Ideally upcoming QGIS will automatically remove libpdal-util13 (and its reverse dependencies). More information in qgis/QGIS#56285 (comment).

[florisvdh]

Wrapping up now that QGIS 3.36 has arrived in the QGIS 'ubuntugis' repo.

The qgis_process backend crashes if an old PDAL plugin (draco) is still installed, which QGIS does not depend on anymore; it makes the R package unusable.

The problem, if present, can be verified by running some example code in a shell:

$ qgis_process plugins 1> /dev/null
double free or corruption (!prev)
Aborted (core dumped)
$ 
$ qgis_process help native:buffer 1> /dev/null
double free or corruption (!prev)
Aborted (core dumped)

If you have this problem, an efficient solution is to purge libpdal-util13:

$ apt purge libpdal-util13 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  libpdal-base13* libpdal-plugin-draco* libpdal-util13*
0 upgraded, 0 newly installed, 3 to remove and 0 not upgraded.
After this operation, 6410 kB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 571982 files and directories currently installed.)
Removing libpdal-plugin-draco (2.5.0+ds-1~jammy3) ...
Removing libpdal-base13 (2.5.0+ds-1~jammy3) ...
Removing libpdal-util13 (2.5.0+ds-1~jammy3) ...
Processing triggers for libc-bin (2.35-0ubuntu3.6) ...

Then the problems should be gone.

$ qgis_process plugins 1> /dev/null
$ qgis_process help native:buffer 1> /dev/null
$ 

Eventually the incompatibility between the more recent PDAL version (which QGIS now depends on) and these older packages should be resolved at some point in the ubuntugis-unstable PPA.

Closing this issue since the workaround is clear and the upstream problems can be tracked in qgis/QGIS#56285 and https://trac.osgeo.org/ubuntugis/ticket/87.