Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Infinite loop in Tree Building Kernel #17

Closed
rmrao opened this issue Sep 3, 2018 · 1 comment
Closed

Infinite loop in Tree Building Kernel #17

rmrao opened this issue Sep 3, 2018 · 1 comment
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@rmrao
Copy link
Collaborator

rmrao commented Sep 3, 2018

This is a bug that has been present for a while in tsne-cuda and that we can't seem to track down. It appears to occur when the number of nodes allocated for the barnes-hut tree is exceeded by the tree building kernel. The way this is handled inside the code at the moment causes an infinite loop.

First of all - this should be impossible. Unless there are two points in exactly the same position, then it only takes 2N tree nodes to separate all the data perfectly. We've checked, and there aren't two points in exactly the same position.

Furthermore, increasing the number of allocated nodes only delays the problem, and doesn't solve it. Printing out the number of used nodes shows that this is below 2N nodes, and well below the new increased number of nodes, in the iteration before the infinite loop.

The bug also appears data/learning rate dependent. Some combinations of datasets, perplexities, and learning rates cause the bug, while others do not. It is at least partially deterministic because the same combination of dataset and input parameters will cause the bug in the same location. However, saving the state of the program (the current embedded positions, the input data, learning rate, etc.) and restarting it at that point does not seem to cause the bug.

@rmrao
Copy link
Collaborator Author

rmrao commented Jun 11, 2019

Fixed by #39

@rmrao rmrao closed this as completed Jun 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants