Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pdal pipeline crash when using filters.csf and filters.covariancefeatures #3890

Open
ClementAlba opened this issue Nov 17, 2022 · 2 comments
Open

Pdal pipeline crash when using filters.csf and filters.covariancefeatures #3890

ClementAlba opened this issue Nov 17, 2022 · 2 comments

Comments

@ClementAlba
Copy link

ClementAlba commented Nov 17, 2022 •
edited

Hi everyone,

I'm encountering problems with filters.csf and filters.covariancefeatures.
A little bit of context : I had a scan separate in 8 tiles (but for this example, let's say we only have to, it will be easier). First, I run a first pipeline on these tiles to add the OriginId dimension for each point to identify which point correspond to which scan :

[
  {
    "type": "readers.las",
    "filename": "..."
  },
  {
    "type": "filters.python",
    "script": "D:/calba/street_pointcloud_process/src/terrestre/seg_class_mobilier/filters/marquage_obj_mobiles.py",
    "function": "marquage",
    "module": "anything",
    "add_dimension": "OriginId"
  },
  {
    "type": "writers.las",
    "filename": "...",
    "extra_dims": "all",
    "minor_version": 4
  }
]

import numpy as np

def marquage(ins,outs):

    n_points = len(ins['X'])

    identifiant_scan = ins['Intensity'][10] + ins['X'][10] + ins['Y'][10] + ins['Z'][10]
    
    for point in range(0, n_points):
        ins['OriginId'][point] = identifiant_scan
    
    outs['OriginId'] = ins['OriginId']
    
    return True

I get these two files in output :

Then, I want to process this pipeline on these two files :

[
	{
                "type" : "readers.las",
		"filename": "..."
        },
	{
                "type":"filters.csf",
                "resolution":0.5,
		"threshold":0.5,
		"smooth":"true",
		"step":0.4,
		"rigidness":2,
		"iterations":500
        },
	{
                 "type" : "filters.neighborclassifier",
                 "domain" : "Classification[1:2]",
                 "k" : 50
        },
	{
                "type":"filters.covariancefeatures",
		"where": "Classification == 2",
                "radius": 0.05,
                "feature_set": "Verticality,Anisotropy"
        },
	{
		"type": "writers.las",
		"filename": "..."
	}
]

But there is a problem, the pipeline is ok on the wam-ob00000.las file but not on wam-ob00004.las, on this file the pipeline crash and stop.

After some researches I think the filers.covariancefeatures is responsible of the bug, but I really don't know why it is not working on the second file. I checked the pdal info output for wam-ob00004.las and nothing weird.

Did someone have an idea ?

System/installation information:

PDAL version :

pdal 2.4.3 (git-version: Release)

ver :

Microsoft Windows [version 10.0.19044.2251]

Conda details :

$ conda list

bleach                    5.0.1              pyhd8ed1ab_0    conda-forge
blosc                     1.21.1               h74325e0_3    conda-forge
bokeh                     3.0.2              pyhd8ed1ab_0    conda-forge
boost-cpp                 1.78.0               h9f4b32c_1    conda-forge
brotli                    1.0.9                hcfcfb64_8    conda-forge
brotli-bin                1.0.9                hcfcfb64_8    conda-forge
brotlipy                  0.7.0           py39ha55989b_1005    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2022.9.24            h5b45459_0    conda-forge
cairo                     1.16.0            hd694305_1014    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py39h68f70e3_2    conda-forge
cfitsio                   4.1.0                h5a969a9_0    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
click                     8.1.3           win_pyhd8ed1ab_2    conda-forge
cloudpickle               2.2.0              pyhd8ed1ab_0    conda-forge
cmarkgfm                  0.8.0            py39ha55989b_2    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
commonmark                0.9.1                      py_0    conda-forge
contourpy                 1.0.6            py39h1f6ef14_0    conda-forge
cryptography              38.0.3           py39h58e9bdb_0    conda-forge
curl                      7.86.0               heaf79c2_1    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.12.0           py39ha55989b_1    conda-forge
dask                      2022.11.0          pyhd8ed1ab_0    conda-forge
dask-core                 2022.11.0          pyhd8ed1ab_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
distributed               2022.11.0          pyhd8ed1ab_0    conda-forge
docutils                  0.19             py39hcbf5309_1    conda-forge
draco                     1.5.5                h181d51b_0    conda-forge
eigen                     3.4.0                h2d74725_0    conda-forge
expat                     2.5.0                h1537add_0    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.1               hbde0cde_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.38.0           py39ha55989b_1    conda-forge
freetype                  2.12.1               h546665d_0    conda-forge
freexl                    1.0.6                h67ca5e6_1    conda-forge
fsspec                    2022.11.0          pyhd8ed1ab_0    conda-forge
future                    0.18.2             pyhd8ed1ab_6    conda-forge
geos                      3.11.1               h1537add_0    conda-forge
geotiff                   1.7.1                h4ffd875_4    conda-forge
gettext                   0.21.1               h5728263_0    conda-forge
glib                      2.74.1               h12be248_1    conda-forge
glib-tools                2.74.1               h12be248_1    conda-forge
gst-plugins-base          1.21.1               h001b923_1    conda-forge
gstreamer                 1.21.1               h6b5321d_1    conda-forge
hdf4                      4.2.15               h1b1b6ef_5    conda-forge
hdf5                      1.12.2          nompi_h2a0e4a3_100    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
icu                       70.1                 h0e60522_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        5.0.0              pyha770c72_1    conda-forge
importlib_metadata        5.0.0                hd8ed1ab_1    conda-forge
intel-openmp              2022.1.0          h57928b3_3787    conda-forge
jaraco.classes            3.2.3              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jpeg                      9e                   h8ffe710_2    conda-forge
kealib                    1.4.15               hdf81f3a_1    conda-forge
keyring                   23.11.0          py39hcbf5309_0    conda-forge
kiwisolver                1.4.4            py39h1f6ef14_1    conda-forge
krb5                      1.19.3               h1176d77_0    conda-forge
lcms2                     2.14                 h90d422f_0    conda-forge
lerc                      4.0.0                h63175ca_0    conda-forge
libblas                   3.9.0              16_win64_mkl    conda-forge
libbrotlicommon           1.0.9                hcfcfb64_8    conda-forge
libbrotlidec              1.0.9                hcfcfb64_8    conda-forge
libbrotlienc              1.0.9                hcfcfb64_8    conda-forge
libcblas                  3.9.0              16_win64_mkl    conda-forge
libclang                  15.0.4          default_h77d9078_0    conda-forge
libclang13                15.0.4          default_h77d9078_0    conda-forge
libcurl                   7.86.0               heaf79c2_1    conda-forge
libdeflate                1.14                 hcfcfb64_0    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libgdal                   3.6.0                h6176ca1_2    conda-forge
libglib                   2.74.1               he8f3873_1    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
libkml                    1.3.0             hf2ab4e4_1015    conda-forge
liblapack                 3.9.0              16_win64_mkl    conda-forge
libnetcdf                 4.8.1           nompi_h8c042bf_106    conda-forge
libogg                    1.3.4                h8ffe710_1    conda-forge
libpng                    1.6.38               h19919ed_0    conda-forge
libpq                     14.5                 h04fd8be_1    conda-forge
librttopo                 1.1.0               he22b5cd_12    conda-forge
libspatialite             5.0.1               h07bf483_22    conda-forge
libsqlite                 3.39.4               hcfcfb64_0    conda-forge
libssh2                   1.10.0               h680486a_3    conda-forge
libtiff                   4.4.0                h8e97e67_4    conda-forge
libvorbis                 1.3.7                h0e60522_0    conda-forge
libwebp-base              1.2.4                h8ffe710_0    conda-forge
libxcb                    1.13              hcd874cb_1004    conda-forge
libxml2                   2.10.3               hc3477c8_0    conda-forge
libzip                    1.9.2                hfed4ece_1    conda-forge
libzlib                   1.2.13               hcfcfb64_4    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       4.0.2            py39hf617134_0    conda-forge
lz4-c                     1.9.3                h8ffe710_1    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
markupsafe                2.1.1            py39ha55989b_2    conda-forge
matplotlib                3.6.2            py39hcbf5309_0    conda-forge
matplotlib-base           3.6.2            py39haf65ace_0    conda-forge
mkl                       2022.1.0           h6a75c08_874    conda-forge
more-itertools            9.0.0              pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.4            py39h1f6ef14_1    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
nitro                     2.7.dev6             h39d44d4_5    conda-forge
numpy                     1.23.4           py39hbccbffa_1    conda-forge
openjpeg                  2.5.0                hc9384bd_1    conda-forge
openssl                   1.1.1s               hcfcfb64_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.5.1            py39h2ba5b7c_1    conda-forge
partd                     1.3.0              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h0e60522_0    conda-forge
pcre2                     10.40                h17e33f8_0    conda-forge
pdal                      3.2.0                    pypi_0    pypi
pdal-parallelizer         1.10.19                  py39_0    clementalba
pillow                    9.2.0            py39h595c93f_3    conda-forge
pip                       22.3.1             pyhd8ed1ab_0    conda-forge
pixman                    0.40.0               h8ffe710_0    conda-forge
pkginfo                   1.8.3              pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
poppler                   22.11.0              ha6c1112_0    conda-forge
poppler-data              0.4.11               hd8ed1ab_0    conda-forge
postgresql                14.5                 h392df6f_1    conda-forge
proj                      9.1.0                h3863b3b_0    conda-forge
psutil                    5.9.4            py39ha55989b_0    conda-forge
pthread-stubs             0.4               hcd874cb_1001    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.13.0             pyhd8ed1ab_0    conda-forge
pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyproj                    3.4.0            py39h9727d73_2    conda-forge
pyqt                      5.15.7           py39hb77abff_2    conda-forge
pyqt5-sip                 12.11.0          py39h99910a6_2    conda-forge
pysocks                   1.7.1              pyh0701188_6    conda-forge
python                    3.9.13          h9a09f29_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-pdal               3.2.0            py39h97fe608_2    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2022.6             pyhd8ed1ab_0    conda-forge
pywin32-ctypes            0.2.0           py39hcbf5309_1006    conda-forge
pyyaml                    6.0              py39ha55989b_5    conda-forge
qt-main                   5.15.6               h9c3277a_1    conda-forge
readme_renderer           37.3               pyhd8ed1ab_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
requests-toolbelt         0.10.1             pyhd8ed1ab_0    conda-forge
rfc3986                   2.0.0              pyhd8ed1ab_0    conda-forge
rich                      12.6.0             pyhd8ed1ab_0    conda-forge
setuptools                65.5.1             pyhd8ed1ab_0    conda-forge
sip                       6.7.4            py39h99910a6_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.9                hfb803bf_2    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlite                    3.39.4               hcfcfb64_0    conda-forge
tbb                       2021.7.0             h91493d7_0    conda-forge
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
tiledb                    2.11.3               h5689973_1    conda-forge
tk                        8.6.12               h8ffe710_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py39hb82d6ee_3    conda-forge
twine                     4.0.1              pyhd8ed1ab_1    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022f                h191b570_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
unicodedata2              15.0.0           py39ha55989b_0    conda-forge
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
vc                        14.3                 h3d8a991_9    conda-forge
vs2015_runtime            14.32.31332          h1d6e394_9    conda-forge
webencodings              0.5.1                      py_1    conda-forge
wheel                     0.38.4             pyhd8ed1ab_0    conda-forge
win_inet_pton             1.1.0              pyhd8ed1ab_6    conda-forge
xerces-c                  3.2.4                h63175ca_1    conda-forge
xorg-libxau               1.0.9                hcd874cb_0    conda-forge
xorg-libxdmcp             1.1.3                hcd874cb_0    conda-forge
xyzservices               2022.9.0           pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
yaml                      0.2.5                h8ffe710_2    conda-forge
zict                      2.2.0              pyhd8ed1ab_0    conda-forge
zipp                      3.10.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hcfcfb64_4    conda-forge
zstd                      1.5.2                h7755175_4    conda-forge

$ conda info

     active environment : tools_install_test
    active env location : D:\applications\miniconda3\envs\tools_install_test
            shell level : 2
       user config file : C:\Users\calba\.condarc
 populated config files : C:\Users\calba\.condarc
          conda version : 22.9.0
    conda-build version : not installed
         python version : 3.9.12.final.0
       virtual packages : __cuda=11.4=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : D:\applications\miniconda3  (writable)
      conda av data dir : D:\applications\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : D:\applications\miniconda3\pkgs
                          C:\Users\calba\.conda\pkgs
                          C:\Users\calba\AppData\Local\conda\conda\pkgs
       envs directories : D:\applications\miniconda3\envs
                          C:\Users\calba\.conda\envs
                          C:\Users\calba\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/22.9.0 requests/2.28.1 CPython/3.9.12 Windows/10 Windows/10.0.19044
          administrator : False
             netrc file : None
           offline mode : False
@chambbj
Copy link
Member

chambbj commented Dec 5, 2022

Looks like a couple of issue going on here. First, the eigen decomposition to obtain covariance features is failing. I wonder if a larger radius could help to get around this (or use fixed knn rather than radius). Second, we are not cleanly exiting all threads when the exception is thrown about the eigenvalues.

@ClementAlba
Copy link
Author

Hi @chambbj, thanks for your response.

I searched in the source code of the covariancefeatures filter and found this about the eigen decomposition :

PDAL/filters/CovarianceFeaturesFilter.cpp

Line 222 in a00819d

if (solver.info() != Success)

What does the solver.info() check ? Because if the solver.info() is not equal to "Success", I will see the message Cannot perform eigen decomposition in my console right ? So I wonder if the eigen decomposition gives "corrupted" values or something like that.

And for the threads who not exit cleanly, do you know which exception is thrown ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chambbj @ClementAlba