Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error using fnearneigh_gpu #19

Open
dpdarrow opened this issue Jul 21, 2016 · 6 comments
Open

Error using fnearneigh_gpu #19

dpdarrow opened this issue Jul 21, 2016 · 6 comments

Comments

@dpdarrow
Copy link

I am working on a huge dataset and having trouble getting the gpu script working. I was able to solve library issues, but I am stuck on this error:

TEprepare.m - line 934: Optimal tau for this dataset is: 0.39
TEprepare.m - line 946: Optimal dimension for this dataset is: 8
TEprepare.m - line 1056: Adding TEprepare structure to original data structure

InteractionDelayReconstruction_calculate.m - line 113:

############# OPTIMIZING INFORMATION TRANSFER DELAY
  TEfindDelay.m - line 45: Estimating TE for u = 40 ms

Error using fnearneigh_gpu
Error detected in the GPU (check console)

Error in TEcallGPUsearch (line 295)
[~, distance_p21] = fnearneigh_gpu(single(pointset_p21),single(pointset_p21),k_th,TheilerT,nchunks);

...

The error reported in the console is:
invalid device function

I tried recompiling with the latest nvidia cuda, but no luck so far

@trentool
Copy link
Owner

trentool commented Aug 2, 2016

Hi @dpdarrow ,
this seems to be a problem with your device, from googling around, I got that you may have to specify a different architecture when compiling the CUDA functions (-arch=compute_xx and -code=sm_xx). Did you try to change these settings when recompiling (you should be able to set them in the CUDA Makefile)?

@dpdarrow
Copy link
Author

Thanks. I was able to find improvements from passing the compute flags. I realized that I am getting errors from running make in the cuda directory. Unfortunately, I am getting a reference error:

/usr/bin/g++ -c -DMX_COMPAT_32 -D_GNU_SOURCE -DMATLAB_MEX_FILE -I"/usr/local/MATLAB/R2016a/extern/include" -I"/usr/local/MATLAB/R2016a/simulink/include" -ansi -fexceptions -fPIC -fno-omit-frame-pointer -pthread -std=c++11 -O -DNDEBUG /home/dpdarrow/matlab/TRENTOOL3/cuda/fnearneigh_gpu.cpp -o /tmp/mex_252588407813488_30845/fnearneigh_gpu.o
/usr/bin/g++ -pthread -Wl,--no-undefined -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2016a/extern/lib/glnxa64/mexFunction.map" /tmp/mex_252588407813488_30845/fnearneigh_gpu.o -lgpuKnnLibrary -lcudart -lcusparse -lcublas -L. -L/usr/local/cuda/lib64 -Wl,-rpath-link,/usr/local/MATLAB/R2016a/bin/glnxa64 -L"/usr/local/MATLAB/R2016a/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o fnearneigh_gpu.mexa64
/tmp/mex_252588407813488_30845/fnearneigh_gpu.o: In function mexFunction': fnearneigh_gpu.cpp:(.text+0x249): undefined reference tocudaFindKnn(int_, float_, float_, float_, int, int, int, int, int)'
collect2: error: ld returned 1 exit status

I was able to adjust the compute code to 52 but this error arises either way. Thank you for your help! I will continue to attempt other solutions.

@dpdarrow
Copy link
Author

I saw the other post by @samuelandjw and found his linking fix of removing extern C from gpuKnnLibrary.cu, which seemed to solve the compilation issue.
Unfortunately, the gpu run has been seg faulting matlab, so I am hoping to track that down.

@dpdarrow
Copy link
Author

I finally determined that the major issue was kernel interruption from the OS on a nondedicated GPU last year. I upgraded to a new computer, and unfortunately, after fixing everything that had been a problem in the past, I am now receiving the same error.

TEfindDelay.m - line 45: Estimating TE for u = 10 msError using fnearneigh_gpu
Error detected in the GPU (check console)

Error in TEcallGPUsearch (line 313)
[~, distance_p21] = fnearneigh_gpu(single(pointset_p21),single(pointset_p21),k_th,TheilerT,nchunks);

Error in TEsurrogatestats_ensemble (line 880)
[ncount] = TEcallGPUsearch(cfg,channelpair,pointsets_concat_1,pointsets_concat_2, ...

Error in TEfindDelay (line 57)
TGA_results{uu}=TEsurrogatestats_ensemble(cfgTESS,data);

Error in InteractionDelayReconstruction_calculate (line 115)
[dataprep, TEmat] =

TEfindDelay(predicttimevec_u,cfgTESS,dataprep);

console: invalid device function

current cuda library is 8 and the dedicated Gpu is P5000

Any thoughts or help would be greatly appreciated

@mwibral
Copy link
Collaborator

mwibral commented Oct 13, 2017 via email

@Xirailuyo
Copy link

See my answer here: #27

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants