Error using gpuArray/nan on unpadded data

[thosvarley]

I have a set of 51 essentially identical recordings (recorded on the same array, same electrode layout, each one is 60 minutes exactly), but a subset of them throw the following error. I have searched for this error before and have seen it has to do with adding padding between trials, but that does not happen here. We are only looking at continuous recordings.

We are running Kilosort2.5 on a Red Hat Enterprise Linux supercomputing cluster.

I cannot figure out what makes some files different (resulting in this error).

                            < M A T L A B (R) >
                  Copyright 1984-2022 The MathWorks, Inc.
             R2022a Update 2 (9.12.0.1956245) 64-bit (glnxa64)
                                May 11, 2022

 
To get started, type doc.
For product information, visit www.mathworks.com.
 
[�Warning: Name is nonexistent or not a directory:
/geode2/home/u100/tvarley/Carbonate/Libraries/spike_sorting/npy-matlab]� 
[�> In path (line 109)
In addpath (line 86)
In main_kilosort_func (line 6)]� 
Looking for data inside /N/project/dpt_spikes/data/5-12-21/Slice3/03_WASHOUT/data000/_16bit 
Time   0s. Computing whitening matrix.. 
Getting channel whitening matrix... 
Channel-whitening matrix computed. 
Time  49s. Loading raw data and applying filters... 
Time 1027s. Finished preprocessing 1098 batches. 
pitch is 60 um
0.35 sec, 1 batches, 10000 spikes 
24.95 sec, 101 batches, 1010000 spikes 
49.19 sec, 201 batches, 1951661 spikes 
73.44 sec, 301 batches, 2870023 spikes 
97.74 sec, 401 batches, 3720266 spikes 
122.07 sec, 501 batches, 4565340 spikes 
146.46 sec, 601 batches, 5257011 spikes 
170.80 sec, 701 batches, 6076888 spikes 
195.02 sec, 801 batches, 6891565 spikes 
220.52 sec, 901 batches, 7597331 spikes 
243.27 sec, 1001 batches, 8543765 spikes 
265.52 sec, 1098 batches, 9237992 spikes 
time 651.34, Shifted up/down 1098 batches. 
Time 660s. Optimizing templates ...
{�Error using gpuArray/nan
An unexpected error occurred trying to launch a kernel. The CUDA error was:
invalid configuration argument

Error in median (line 71)
        y = nan(s,'like',x);

Error in learnTemplates (line 248)
            toc, ibatch, niter, Nfilt, sum(nsp), median(mu), numel(st0), ndrop)

Error in learnAndSolve8b (line 35)
    rez     = learnTemplates(rez, rez.iorig);

Error in main_kilosort_func (line 47)
rez = learnAndSolve8b(rez, iseed);
}� 

[dmosdallas]

I've been running into the same issue (unconcatenated recordings with no periods of inactivity, some recordings run fine, others get this error).

I've noticed that reducing ops.fshigh to something like 150Hz makes this failure less likely, but I'm still not sure what the underlying cause is here.

[dmosdallas]

I think I've figured out what was causing this issue with the data I was running.

We're using a 64 channel linear probe and the most superficial channels have relatively few spikes. By default, it looks like data is segmented into blocks (ops.nblocks, 5 by default) of contiguous channels for motion correction (?) before trying to pick out clusters.

I think this means that if there's a batch where it's not seeing spikes in one of this blocks, we'll get this crash. I've tried re-running kilosort on the previously-failing data with ops.nblocks = 0 and I'm no longer getting these crashes.

Hope this helps!