Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[visualize normal] Normals are displayed incorrectly #6024

Closed
QiuYilin opened this issue Apr 29, 2024 · 19 comments · Fixed by #6036
Closed

[visualize normal] Normals are displayed incorrectly #6024

QiuYilin opened this issue Apr 29, 2024 · 19 comments · Fixed by #6036
Labels

Comments

@QiuYilin
Copy link

QiuYilin commented Apr 29, 2024

operating system: Windows 10/Windows 11
PCL version: 1.14.0/1.14.1 fc54abc
/8e2d2839fe73a955d49b9a72861de2becf2da
compiler:MSVC 2022 17.8.34309/17.9.5

Details of the environment where I had the problem:
vcpkg baseline: 326d8b43e365352ba3ccadf388d989082fe0f2a6
vcpkg install pcl[qt,vtk,cuda]

The pcl (1.13.1 without cuda) in my old version of vcpkg is correct.

I want to visualize the normals of a point cloud, but the display looks weird.
image

My code:

#define PCL_NO_PRECOMPILE

#include <pcl/features/normal_3d_omp.h>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <pcl/visualization/pcl_visualizer.h>

#include <iostream>

int main() {

  pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
  if (pcl::io::loadPCDFile<pcl::PointXYZ>(
          "D:/program_on_git/own/test/pcl_test/source/croped.pcd", *cloud) ==
      -1) {
    PCL_ERROR("Couldn't read file.\n");
    return -1;
  }

  pcl::NormalEstimationOMP<pcl::PointXYZ, pcl::Normal> ne;
  ne.setInputCloud(cloud);
  ne.setNumberOfThreads(8);
  pcl::search::KdTree<pcl::PointXYZ>::Ptr tree(
      new pcl::search::KdTree<pcl::PointXYZ>());
  ne.setSearchMethod(tree);
  pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
  ne.setKSearch(100); 
  ne.setViewPoint(100, 100, 1000);
  ne.compute(*normals);


  pcl::visualization::PCLVisualizer viewer("Point Cloud Viewer");
  viewer.setBackgroundColor(0.0, 0.0, 0.0);
  viewer.addPointCloud<pcl::PointXYZ>(cloud, "cloud");
  viewer.addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(
      cloud, normals, cloud->size() / 100, 20, "normals");
  viewer.setPointCloudRenderingProperties(
      pcl::visualization::PCL_VISUALIZER_POINT_SIZE, 1, "cloud");
  viewer.spin();

  return 0;
}

The source cloud has 1280000 points. If I extract 432 keypoints from it ,the normals computed will be correct

image

@QiuYilin QiuYilin added kind: bug Type of issue status: triage Labels incomplete labels Apr 29, 2024
@larshg
Copy link
Contributor

larshg commented Apr 29, 2024

Could you share the pointcloud?
I have visualized quite a few pointclouds over time and it worked well - so I doubt its a bug in PCL.

@larshg larshg added kind: question Type of issue and removed kind: bug Type of issue status: triage Labels incomplete labels Apr 29, 2024
@QiuYilin
Copy link
Author

@mvieth
Copy link
Member

mvieth commented Apr 29, 2024

  • Which operating system do you use?
  • Which PCL version do you use? How did you install it?
  • Which compiler do you use?

@QiuYilin
Copy link
Author

@mvieth Sorry I forget to say that.
operating system: Windows 11
PCL version: 1.14.0 fc54abc
compiler:MSVC 2022

@larshg
Copy link
Contributor

larshg commented Apr 29, 2024

Tested it on ubuntu rq:
image

@QiuYilin
Copy link
Author

QiuYilin commented Apr 29, 2024

@larshg I found that after removing the invalid points, it was displayed normally, but I did not see any instructions or errors that prohibit the use of invalid points for normal estimation.Why not process it into an algorithm that can filter out invalid points on the underlying mechanism, or are invalid points meaningful in some cases(It is true that invalid values are used when keeping organized, so why not let all classes accept this ordered structure?)? At present, I have a vague feeling that some classes are sensitive to invalid points and some are not. Is there any unified guiding principle?

@larshg
Copy link
Contributor

larshg commented Apr 29, 2024

Do you build in release or debug - I can verify that the normals is wrong when running in debug mode for some (probably memory indexing) reason.

Not sure if its due to invalid points though - I can try add that to verify.

@larshg larshg added kind: bug Type of issue platform: windows and removed kind: question Type of issue labels Apr 29, 2024
@mvieth
Copy link
Member

mvieth commented Apr 29, 2024

For normal estimation, invalid points should not be a problem. The normal for an invalid point will simply be (NaN, NaN, NaN).
And addPointCloudNormals also checks whether point and normal are valid: https://github.com/PointCloudLibrary/pcl/blob/master/visualization/include/pcl/visualization/impl/pcl_visualizer.hpp#L909
You could also test what happens if you pretend that your point cloud is unorganized, by adding this after normal estimation:

cloud->width = cloud->width * cloud->height;
cloud->height = 1;
normals->width = normals->width * normals->height;
normals->height = 1;

@QiuYilin
Copy link
Author

QiuYilin commented Apr 30, 2024

@mvieth Modified code pretending unorganized:

#define PCL_NO_PRECOMPILE

#include <pcl/features/normal_3d_omp.h>
#include <pcl/filters/filter_indices.h>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <pcl/visualization/pcl_visualizer.h>

#include <iostream>

int main() {
  pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
  if (pcl::io::loadPCDFile<pcl::PointXYZ>(
          "D:/program_on_git/own/test/pcl_test/source/croped.pcd", *cloud) ==
      -1) {
    PCL_ERROR("Couldn't read file.\n");
    return -1;
  }

  //   std::vector<int> vec;
  //   pcl::removeNaNFromPointCloud(*cloud, *cloud, vec);

  pcl::NormalEstimationOMP<pcl::PointXYZ, pcl::Normal> ne;
  ne.setInputCloud(cloud);
  ne.setNumberOfThreads(8);
  pcl::search::KdTree<pcl::PointXYZ>::Ptr tree(
      new pcl::search::KdTree<pcl::PointXYZ>());
  ne.setSearchMethod(tree);
  pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
  ne.setKSearch(100);
  ne.setViewPoint(100, 100, 1000);
  ne.compute(*normals);

  cloud->width = cloud->width * cloud->height;
  cloud->height = 1;
  normals->width = normals->width * normals->height;
  normals->height = 1;

  pcl::visualization::PCLVisualizer viewer("Point Cloud Viewer");
  viewer.setBackgroundColor(0.0, 0.0, 0.0);
  viewer.addPointCloud<pcl::PointXYZ>(cloud, "cloud");
  viewer.addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(
      cloud, normals, cloud->size() / 100, 20, "normals");
  viewer.setPointCloudRenderingProperties(
      pcl::visualization::PCL_VISUALIZER_POINT_SIZE, 1, "cloud");
  viewer.spin();

  return 0;
}

display: (no normals)
image

@QiuYilin
Copy link
Author

@larshg I build in debug.

@QiuYilin
Copy link
Author

I noticed that if I use BilateralFilter on this point cloud, if the invalid points are not removed, the program will report an error. Is this a feature of BilateralFilter or a related bug?

Mycode:

#include <pcl/filters/bilateral.h>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <pcl/search/kdtree.h>

#include <iostream>

int main() {
  pcl::PointCloud<pcl::PointXYZI>::Ptr cloud(
      new pcl::PointCloud<pcl::PointXYZI>);
  if (pcl::io::loadPCDFile<pcl::PointXYZI>(
          "D:/program_on_git/own/test/pcl_test/source/croped.pcd", *cloud) ==
      -1) {
    PCL_ERROR("Couldn't read input file.\n");
    return -1;
  }

  // if (cloud->is_dense == false) {
  //   std::cout << "Input cloud is not dense." << std::endl;
  // }

  // std::vector<int> vec;
  // pcl::removeNaNFromPointCloud(*cloud, *cloud, vec);

  pcl::BilateralFilter<pcl::PointXYZI> bilateral_filter;
  bilateral_filter.setInputCloud(cloud);
  auto tree = std::make_shared<pcl::search::KdTree<pcl::PointXYZI>>();
  bilateral_filter.setSearchMethod(tree);

  bilateral_filter.setHalfSize(0.05);
  bilateral_filter.setStdDev(0.1);

  pcl::PointCloud<pcl::PointXYZI>::Ptr filtered_cloud(
      new pcl::PointCloud<pcl::PointXYZI>);
  bilateral_filter.filter(*filtered_cloud);

  pcl::io::savePCDFileASCII("filtered_cloud.pcd", *filtered_cloud);
  std::cout << "Filtered cloud saved." << std::endl;

  return 0;
}

report

Assertion failed: point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!", file C:\src\vcpkg\buildtrees\pcl\src\head\05601a9476-82ae3636f9.clean\kdtree\include\pcl/kdtree/impl/kdtree_flann.hpp, line 376
Debug Error!

Program: ...ram_on_git\own\test\pcl_test\build\Debug\bilateral_filter.exe

abort() has been called

(Press Retry to debug the application)

@mvieth
Copy link
Member

mvieth commented Apr 30, 2024

I noticed that if I use BilateralFilter on this point cloud, if the invalid points are not removed, the program will report an error. Is this a feature of BilateralFilter or a related bug?

BilateralFilter should check for invalid points before calling radiusSearch. So yes, that is a bug in BilateralFilter.

@QiuYilin
Copy link
Author

QiuYilin commented May 6, 2024

I tried updating the version of vcpkg and recompiling pcl, but the problem still didn't solve.

@QiuYilin
Copy link
Author

QiuYilin commented May 7, 2024

The normals can be displayed correctly in pcl 1.13.1. However,new problems about "#define PCL_NO_PRECOMPILE" were generated:

[build] lld-link : error : undefined symbol: omp_get_num_procs [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:52
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::setNumberOfThreads(unsigned int))
[build]   
[build] lld-link : error : undefined symbol: __kmpc_global_thread_num [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:63
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:310
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: virtual int __cdecl flann::NNIndex<struct flann::L2_Simple<float>>::knnSearch(class flann::Matrix<float> const &, class flann::Matrix<unsigned __int64> &, class flann::Matrix<float> &, unsigned __int64, struct flann::SearchParams const &) const)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\lsh_index.h:229
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: virtual int __cdecl flann::LshIndex<struct flann::L2_Simple<float>>::knnSearch(class flann::Matrix<float> const &, class flann::Matrix<unsigned __int64> &, class flann::Matrix<float> &, unsigned __int64, struct flann::SearchParams const &) const)
[build]   >>> referenced 1 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_push_num_threads [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:73
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:102
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:327
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: virtual int __cdecl flann::NNIndex<struct flann::L2_Simple<float>>::knnSearch(class flann::Matrix<float> const &, class flann::Matrix<unsigned __int64> &, class flann::Matrix<float> &, unsigned __int64, struct flann::SearchParams const &) const)
[build]   >>> referenced 6 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_fork_call [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:73
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:102
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:327
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: virtual int __cdecl flann::NNIndex<struct flann::L2_Simple<float>>::knnSearch(class flann::Matrix<float> const &, class flann::Matrix<unsigned __int64> &, class flann::Matrix<float> &, unsigned __int64, struct flann::SearchParams const &) const)
[build]   >>> referenced 6 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_for_static_init_8 [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:79
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:108
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.1)
[build]   
[build] lld-link : error : undefined symbol: __kmpc_for_static_fini [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:73
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:102
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.1)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced 6 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_for_static_init_4 [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:345
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.5)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\lsh_index.h:241
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.8)
[build]   >>> referenced 4 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_reduce [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:345
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.5)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\lsh_index.h:241
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.8)
[build]   >>> referenced 4 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_end_reduce [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:345
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.5)
[build]   >>> referenced 11 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_barrier [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:345
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.5)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\lsh_index.h:241
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.8)
[build]   >>> referenced 4 more times

@QiuYilin
Copy link
Author

QiuYilin commented May 8, 2024

At present, I have reverted to a version that can display normally:

Windows 10(Version 22H2 19045.4291)

MSVC 2019(14.29.30133)

vcpkg.json

{
    "dependencies":[
        {
            "name": "pcl",
            "version>=": "1.12.0",
            "features": ["vtk"]
        }
    ],
    "builtin-baseline":"8eb57355a4ffb410a2e94c07b4dca2dffbee8e50",
    "overrides":[
        {
            "name": "pcl",
            "version": "1.13.1#1"
        }
    ]
}

I hope there is a hint that the 1.14.1 version of pcl on windows can run normally.

@mvieth
Copy link
Member

mvieth commented May 9, 2024

Okay, I may have found the problem. As far as I can tell, the normals are only displayed incorrectly if Windows is used, and the debug configuration is used (release works fine), and the cloud or normals given to the visualizer contain invalid values (NaN). As I mentioned earlier, the addPointCloudNormals function does check for invalid values. However, the last few entries of pts are left uninitialized if invalid values are encountered, and when pts is later passed to vtk the full size is given instead of only the number of valid entries in pts. So I think if nr_normals is updated with cell_count or j respectively, it should work correctly. I am not sure why this is only a problem on Windows in debug configuration, but maybe memory allocated with new is filled with special values in that situation?
I will create a pull request soon to fix this.

@QiuYilin
Copy link
Author

If I switch to release, will there be any other problems besides the normal display? I switched to the release mode of 1.14.1 and encountered the following situation: SamplingSurfaceNormal will crash if the parameters are set to 0. Is this normal?

@mvieth
Copy link
Member

mvieth commented May 10, 2024

SamplingSurfaceNormal will crash if the parameters are set to 0. Is this normal?

Which parameter? sample or ratio?

@QiuYilin
Copy link
Author

SamplingSurfaceNormal will crash if the parameters are set to 0. Is this normal?

Which parameter? sample or ratio?

both

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants