Python: tsnecuda.TSNE() error, illegal memory access

[miqueleg]

I'm having some problems in order to execute the tsnecuda.TSNE() in my dataset.
Here you have a link to a github with the extended information and the data in order to reproduce the error:https://github.com/miqueleg/cuda-tSNE_problem
A a summary:
When I execute the TSNE function, a error is displayed:

GPUassert: an illegal memory access was encountered /home/rmrao/miniconda3/conda-bld/tsnecuda_1538693499277/work/src/util/cuda_utils.cu 55

System information:

OS: CentOS Linux 7
GPU: nvidia GeForce GTX980
Python 3.6.3
faiss-cpu 1.4.0 py36_cuda0.0_1 pytorch
faiss-gpu 1.4.0 py36_cuda8.0.61_1 pytorch
tsnecuda 0.1.1 py36_cuda80_0 [cuda80] cannylab
tsnecuda 0.1.1

I hope you can help me!

Thanks,
Miquel Estévez Gay
CompBioLab, UdG

[Hippogriff]

Any update on this issue?

[acletournel]

Hello, I've got exactly the same issue:

GPUassert: an illegal memory access was encountered /home/rmrao/miniconda3/conda-bld/tsnecuda_1538622901232/work/src/util/cuda_utils.cu 55

My environment looks like:

OS: Ubuntu 16.04
GPU: TitanX Pascal 12 Go
Python: 3.6.7
cuda90 1.0 h6433d27_0 pytorch
faiss-gpu 1.4.0 py36_cuda9.0.176_1 [cuda90] pytorch
tsnecuda 0.1.1 py36_0 cannylab

The Install is validated with the random(5000,50) dataset as indicated.

The fit_transform gives a result with a random(5583,80) dataset (size of mine); max value 0.999 and min value 3e-7. Takes 4s.

The fit_transform fails (GPUassert ... 55) with my sample dataset(5583,80); max value 0.967 and min value -0.601.

The fit transform fails (GPUassert ... 55) with my dataset.abs(5583,80); max value 0.967 and min value 8.67e-8.
I don't understand what goes wrong with my sample dataset.

My actual dataset is (785000,200).
The biggest random dataset that works is approx (296k, 200).
With (300k, 200), I get the error

terminate called after throwing an instance of 'thrust::system::system_error'
what(): parallel_for failed: out of memory

I don't understand why the computing time for the projection grows up to 2mn31 for 285k and then decreases to 22.0s for 292k and 22.3s for 296k ??

[rmrao]

Sorry for the long delay in response. There has been a bug that we've been tracking for a long time that causes this problem randomly. We believe we fixed it, although we're not 100% sure. We also have a new branch ("fit_tsne") which implements a version of tSNE using fast fourier transforms. This is both faster and more stable than the current Barnes-Hut implementation.

We will be migrating that to the master branch at some point and also putting it on conda. In the meantime, if you'd like to try cloning the fit_tsne branch and seeing if that works, you may be able to fix the issue that way.

[yts19871111]

This problem still exists. GPUassert: an illegal memory access was encountered /home/rmrao/miniconda3/conda-bld/tsnecuda_1538622901232/work/src/util/cuda_utils.cu 55

[rmrao]

The fit_tsne branch has been migrated to master in #39 and is now part of the new release. This has not been pushed to conda yet, but if you compile from source you shouldn't have any issues.

[rmrao]

Binaries for new release on CUDA 9.0 were just pushed to conda. You can install with conda install tsnecuda cuda90 -c cannylab. This should fix your problem.