Preprocessing error: CUDA_ERROR_ILLEGAL_ADDRESS

[aepadilla4]

Hi everyone,

I am currently trying to sort through some data and run into this error during the preprocessing step. Kilosort only runs for about 3 seconds before this error occurs. I am using MATLAB R2022b with CUDA 12.0 on a Dell Precision 5820 with NVIDIA T400 GPU. Below are the errors that show. I have been trying to work through this for quite some time but haven't been able to figure it out. I appreciate any help!

Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS

In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 732)
In ksGUI>@(,)obj.runPreproc() (line 333)
Warning: Encountered unexpected error during
CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 10)

[gkBCCN]

I also encounter this error, but at different points of the pipeline. The couple times it raised this error in the preprocessing step, I restarted the computer and it solved it, which made me suspect that it was a graphics card issue. However, every time I run the pipeline, I get the same error during trackAndSort(). I am using default parameters and successfully ran the GPU setup in mexGPUall.m. Kilsort 2.5 on the same computer and same data does not have this issue.

[tenorio-vivianesgm]

I also have the same issue! I have the exact same configuration as aepadilla4 except my Matlab is 2022a. Everything else is absolutely the same and I have the same issue.

Any suggestions?

[aepadilla4]

Hi @tenorio-vivianesgm! I was able to resolve this issue by upgrading our current GPU. I found a page that suggested that the GPU may not have enough RAM to support the file size. We upgraded to the NVIDIA T1000 and everything works perfectly. I believe any GPU with at least 8GB of RAM will work. In the meantime, we also explored this new platform (https://www.ontologic.ly) which runs Kilosort and Phy in an online service. It does still have a couple of bugs, but helped us make some progress while we awaited the delivery of our new GPU. Hope this helps!