pdal python filter importing open3d

[gsalgian]

I am trying to use the open3d library in a pdal python filter, but the pipeline crashes when trying to import open3d. Here is a minimal environment to reproduce the problem:

conda create -n pdal-python-test
conda activate pdal-python-test
conda install -c open3d-admin open3d
conda install -c conda-forge pdal python-pdal gdal

Attached is a zipped yml file for this environment: environment.zip

Here are the python filter (the multiply example from the pdal webpage with the import statement added) and the json pipeline file pipeline.zip

And here is the gdb trace: gdb_trace_python_filter.txt

[abellgithub]

Does this work if you don't import open3d?

Do you have using success using conda open3d outside of the PDAL python filter?

[zerotohero]

Some additional info from trying this:
Another requirement appears to be libgl1-mesa-glx.

The test works (generates output) when the "import open3d" line is removed. Just importing the library causes a segfault.

(pdal-python-test) root@d9656dfb08bb:/home# pdal pipeline test.json
munmap_chunk(): invalid pointer
Aborted (core dumped)

[zerotohero]

Additionally, simply importing open3d in python doesn't cause python to segfault. So it appears to be something with pdal importing it via the python filter.

[hobu]

My guess is both PDAL and open3d are mucking with the interpreter in some way, when PDAL awakens its filters.python.

[gsalgian]

Yes, that was what I saw too (the filter works fine without the import open3d statement, but crashes when the import is present). We have been using open3d on its own, that part works too. From: zerotohero ***@***.***> Sent: Tuesday, June 22, 2021 11:35 AM To: PDAL/python ***@***.***> Cc: Garbis Salgian ***@***.***>; Author ***@***.***> Subject: [EXTERNAL] Re: [PDAL/python] pdal python filter importing open3d (#86) Additionally, simply importing open3d in python doesn't cause python to segfault. So it appears to be something with pdal importing it via the python filter. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https:/github.com/PDAL/python/issues/86*issuecomment-866091101__;Iw!!Nv3xtKNH_4uope0!xDISjxm1cnKnQm0BHzL0lF0jVaWk127RGnJ37KYaUY0cKwv4q4ZdPj8eHVX7AJDA$>, or unsubscribe<https://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AUSPKMCZDDGQQDIJZU6MDU3TUCUR3ANCNFSM47D3FE7A__;!!Nv3xtKNH_4uope0!xDISjxm1cnKnQm0BHzL0lF0jVaWk127RGnJ37KYaUY0cKwv4q4ZdPj8eHaissIzX$>.

[gsalgian]

Yes, the pdal python multiply filter works fine without the import open3d statement. Also open3d on its own works fine. From: Andrew Bell ***@***.***> Sent: Tuesday, June 22, 2021 11:22 AM To: PDAL/python ***@***.***> Cc: Garbis Salgian ***@***.***>; Author ***@***.***> Subject: [EXTERNAL] Re: [PDAL/python] pdal python filter importing open3d (#86) Does this work if you don't import open3d? Do you have using success using conda open3d outside of the PDAL python filter? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://urldefense.us/v3/__https:/github.com/PDAL/python/issues/86*issuecomment-866079138__;Iw!!Nv3xtKNH_4uope0!xRyRa_mpBbgY9SJKhxMV38Zzvuv6gGUv5dkb08XgRtWSvBA-nKNXUaCuMeMhYA-w$>, or unsubscribe<https://urldefense.us/v3/__https:/github.com/notifications/unsubscribe-auth/AUSPKMEP4GD5E6KWNBTJ5DLTUCTARANCNFSM47D3FE7A__;!!Nv3xtKNH_4uope0!xRyRa_mpBbgY9SJKhxMV38Zzvuv6gGUv5dkb08XgRtWSvBA-nKNXUaCuMS5efnYM$>.

[abellgithub]

The conda open3d package doesn't seem to work with any version of python past 3.6, though it says it works with up to 3.8. I had success (your test script ran and generated output) with:

$ conda create -n sri -c conda-forge pdal python-pdal python=3.6
$ conda activate sri
$ conda install -c open3d-admin open3d
$ pdal pipeline test.json

Anything later than python 3.6 wouldn't allow import of open3d in python at all:

$ conda create -n open3d -c open3d-admin open3d python=3.7
$ conda activate open3d
$ echo "import open3d" >> foo.py
$ python foo.py
Traceback (most recent call last):
  File "foo.py", line 1, in <module>
    import open3d
  File "/Users/acbell/miniconda3/envs/open3d/lib/python3.7/site-packages/open3d/__init__.py", line 111, in <module>
    import open3d.ml
  File "/Users/acbell/miniconda3/envs/open3d/lib/python3.7/site-packages/open3d/ml/__init__.py", line 35, in <module>
    from . import datasets
  File "/Users/acbell/miniconda3/envs/open3d/lib/python3.7/site-packages/open3d/ml/datasets.py", line 34, in <module>
    from open3d._ml3d.datasets import *
  File "/Users/acbell/miniconda3/envs/open3d/lib/python3.7/site-packages/open3d/_ml3d/datasets/__init__.py", line 5, in <module>
    from .semantickitti import SemanticKITTI
  File "/Users/acbell/miniconda3/envs/open3d/lib/python3.7/site-packages/open3d/_ml3d/datasets/semantickitti.py", line 6, in <module>
    from sklearn.neighbors import KDTree
ModuleNotFoundError: No module named 'sklearn'

Got the same result with python 3.8.
Installing scikit-learn did not help.

Perhaps you should contact the open3d maintainers.

[gsalgian]

I am able to run the latest version of open3d with python 3.8:

(pdal-python-test) e27197@gauss:~$ conda list | grep open3d
open3d                    0.13.0                   py38_0    open3d-admin
(pdal-python-test) e27197@gauss:~$ conda list | grep 'python '
ipython                   7.24.1           py38hd0cf306_0    conda-forge
python                    3.8.10          h49503c6_1_cpython    conda-forge
(pdal-python-test) e27197@gauss:~$ echo "import open3d" >> foo.py
(pdal-python-test) e27197@gauss:~$ python foo.py
(pdal-python-test) e27197@gauss:~$

[abellgithub]

I tried this on docker Ubuntu20 with python 3.6 and had no problems. Is there some reason not to use python 3.6?

[abellgithub]

I figured out what's wrong and it has to do with incompatibility with python libraries WRT PDAL and the way that the extension is loaded. I'm not sure what the best way to fix this is, but I think you can make things work with whatever version of PDAL you like by setting the environment variable PDAL_PYTHON_LIBRARY to the shared python library that corresponds with your python version.

For example export PDAL_PYTHON_LIBRARY=/home/acbell/miniconda3/envs/open3d/lib/libpython3.so

We could certainly set this on conda activation, but I'm not sure beyond that. If you really want to look at the details you can check out PDAL/PDAL#1010, which for the most part I had forgotten.

[abellgithub]

To add... I think normally when people install PDAL via conda they get the latest version of python with which PDAL was built. This makes the shared object consistent with the static library with which the PDAL python library was linked. When they're inconsistent, things can break. I'm really not sure why version 3.6 would work. Perhaps just lucky.

[gsalgian]

Thank you for looking into this. In the conda environment built as described in my original post (first installed open3d, then pdal) I tried export PDAL_PYTHON_LIBRARY=/home/local/SRI/e27197/anaconda3/envs/pdal-python-test/lib/libpython3.so but got the same error as before when trying to run the pipeline:

(pdal-python-test) e27197@gauss:~/testPython$ pdal pipeline test.json
munmap_chunk(): invalid pointer
Aborted (core dumped)

Did you install pdal first and then open3d? Can try that next.

[zerotohero]

With python 3.6 and open3d 0.13.0, I still get the invalid pointer issue as well. Even after the PDAL_PYTHON_LIBRARY export. The only thing that makes it work for me is going back to open3d=0.9.0.

[abellgithub]

I made an error in my testing, so I now have the same failure. Sorry. I'll keep looking.

[zerotohero]

Just keeping everything the same and upgrading from 0.9 to 0.10, that's where things start falling apart for me.