An unexpected error occured trying to launch a kernel. The CUDA error: invalid argument

[Kasia-Bz]

Hello,
I am trying to set up kilosort 2, but I am keep running into the same problem. Kilosort gives me an error: "Error running kilosort! An unexpected error occurred trying to launch a kernel. The CUDA error was: invalid argument.
Saving data to .../filename... Error saving data for phy! Reference to non-existend field "Wphy""

I use Matlab 2019b and CUDA 10.1. I am working with Ubuntu. I tried first to compile the file mexGPUall file and I got response that MEX completed successfully.
The CUDA error appears later in pre-processing process, after the main optimization.
I tried to change CUDA version and update my NVIDIA drivers, but so far that didn't help. Also, I am using NVIDA GPX960 and 32GB memory.
Could you please help me with this issue?

[marius10p]

That's a generic CUDA error whenever something breaks on the GPU. Can you please post your entire error from the command line (not the GUI error), as well as the details of your data (how long, how many channels).

[Kasia-Bz]

Thank you for your reply. Now, my kilosort crashes already on the computing whitening filter step. The error message I get from Matlab:
Unable to find supported GPU device. For more information on GPU support, see GPU Support by Release.

Error in ksGUI/computeWhitening (line 770)
[~,Wrot] = computeWhitening(ops); % this refers to a function outside the gui

Error in ksGUI/updateProbeView (line 1216)
obj.computeWhitening()

Error in ksGUI>@(,)obj.updateProbeView('reset') (line 271)
'Callback', @(,)obj.updateProbeView('reset'));

Error while evaluating UIControl Callback.

I use 385 channels, with a config file for Neuropixel 3B1 version. The file is 85GB, but I also tried to use only part of it of 13GB and I get the same error. I also tried to use data of 66.8GB, which were sorted with Kilosort 2 on someone else's computer, where things worked correctly. When I try to use it on my machine, I get the same error as posted above.

[marius10p]

Ok, you need to make sure first that Matlab can see the GPU correctly. Please run "gpuDevice(1)" at the command line and try to get this to complete successfully. You might need to reinstall graphics drivers.

[Kasia-Bz]

Hello Marius, indeed Matlab could not see the GPU correctly and I kept getting error running "gpuDevice(1)" in command line. I reinstalled CUDA and graphic drivers and now, the error is gone:
"gpuDevice(1)

ans =

CUDADevice with properties:

                  Name: 'GeForce GTX 960'
                 Index: 1
     ComputeCapability: '5.2'
        SupportsDouble: 1
         DriverVersion: 10.2000
        ToolkitVersion: 10.1000
    MaxThreadsPerBlock: 1024
      MaxShmemPerBlock: 49152
    MaxThreadBlockSize: [1024 1024 64]
           MaxGridSize: [2.1475e+09 65535 65535]
             SIMDWidth: 32
           TotalMemory: 2.0981e+09
       AvailableMemory: 1.5368e+09
   MultiprocessorCount: 8
          ClockRateKHz: 1291000
           ComputeMode: 'Default'
  GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
      CanMapHostMemory: 1
       DeviceSupported: 1
        DeviceSelected: 1

"
Yet, I still get an error running Kilosrot2. I am getting the following error:
"Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS

In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
In learnTemplates (line 4)
In learnAndSolve8b (line 12)
In ksGUI/runSpikesort (line 786)
In ksGUI/runAll (line 628)
In ksGUI>@(,)obj.runAll() (line 319)
Error using save
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS

Error in ksGUI/runSaveToPhy (line 828)
save(fname, 'rez', '-v7.3');

Error in ksGUI/runAll (line 629)
obj.runSaveToPhy;

Error in ksGUI>@(,)obj.runAll() (line 319)
'Callback', @(,)obj.runAll());

Error while evaluating UIControl Callback."

[marius10p]

The GPU memory is probably insufficient for sorting Neuropixels data. I would recommend at least 6GB, for example RTX2060 or GTX1660Ti.

[cowenowner]

I had this error on 12-30-2021 with the most recent version of kilosort3 - in extract_spikes.m. This rather kludgy "try/catch" addition to extract_spikes.m seemed to fix the problem for me.

toff = ops.nt0min + t0 + ops.NT *(k-1);
st(1,:) = st(1,:) + toff; 

try
    st(5,:) = cF;
catch
    st(5,:) = cF;
end
st(6,:) = k-1;