Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aborted (core dumped) if I process to many points at once #79

Open
fnardmann opened this issue Sep 3, 2021 · 2 comments
Open

Aborted (core dumped) if I process to many points at once #79

fnardmann opened this issue Sep 3, 2021 · 2 comments

Comments

@fnardmann
Copy link

I plugged the DGCNN model into my semantic segmentation framework in which I use other models like PointNet or PointNet++ without problems. At training time everything is fine and I get pretty good accuracies for my Airborne LiDAR data (here I randomly sample 8192 points for each tile so everything is good). However at test time I want to predict all points inside one tile and I get a memory error for a tile with more than 50000 points.

Aborted (core dumped)

I guess the problem is in the pairwise_distance function. This function calculates a adjacency matrix and I think my gpu memory cant handle an array with the shape of 50000 x 50000. I understand that the tf.matmul function is very fast on gpu but I would like to try a workaround which purely calculates the k nearest neighbors without this huge memory overhead. Is there anything like this? I know how to use a KDTree in normal python but I have not found a way yet to use it with tensorflow placeholders...

I know I can work around this problem by using smaller tiles or downsample my point clouds but I really would like to fix this internally...

I list some basic information about my implementation here:

  • python 3.6
  • tensorflow 1.15
  • batch size is already 1 at test time

Thanks in advance for any tips!

@nazmicancalik
Copy link

I am having a similar issue. Were you able to find a solution ? I would appreciate it if you share it here if you found a way to combat this. Thanks in advance.

@fnardmann
Copy link
Author

Unfortunately I have not found a solution for this. But maybe you could use tf.py_function to build a kd-tree or something like that in plain python.

As an alternative graph-based model I used GACNet and got very good results with it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants