Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

learnTemplates: error indexing into W using gpuArray/subsref on batch with Nfilt=0 #232

Closed
mspacek opened this issue Sep 11, 2020 · 2 comments · Fixed by #595
Closed

learnTemplates: error indexing into W using gpuArray/subsref on batch with Nfilt=0 #232

mspacek opened this issue Sep 11, 2020 · 2 comments · Fixed by #595

Comments

@mspacek
Copy link
Contributor

mspacek commented Sep 11, 2020

After commenting out the print statement that was triggered by a batch with blank raw data with no spikes that was causing the error in #231, I'm now hitting this error slightly later, which seems to be triggered by having Nfilt=0:

>> dbstop if error
>> rez = learnAndSolve8b(rez);
Time 2728s. Optimizing templates ...
Error using gpuArray/subsref
An unexpected error occurred trying to launch a kernel. The CUDA error was:
invalid configuration argument

Error in learnTemplates (line 203)
            W(:,Nfilt + [1:size(dWU0,3)],:) = W0(:,ones(1,size(dWU0,3)),:); % initialize temporal components of waveforms

Error in learnAndSolve8b (line 15)
    rez     = learnTemplates(rez, iorder0);
 
203             W(:,Nfilt + [1:size(dWU0,3)],:) = W0(:,ones(1,size(dWU0,3)),:); % initialize temporal components of waveforms
K>> Nfilt

Nfilt =

     0

K>> size(W)

ans =

    61     0     3

K>> W(:,Nfilt + [1:size(dWU0,3)],:)
Error using gpuArray/subsref
Index exceeds matrix dimension.

K>> dataRAW

dataRAW = <page truncated: showing [1:1455, 1:1] of 65600-by-31>

  65600×31 single gpuArray matrix

    0.0500
   -0.1450
    0.6850
    1.0800
    1.4750
    1.2950
    1.1850
    0.4400
    1.1900    
    ...

K>> nBatches

nBatches =

        9205

K>> ibatch

ibatch =

        3986            

If I print out all the contents of dat (same size as dataRAW), most of it is zeros for this batch, so this is almost certainly at least partially overlapping with the blank periods of data described in #231. Not sure how to proceed.

@JoseGuzman
Copy link
Contributor

JoseGuzman commented Sep 11, 2020

I'm trying to look into this (look in issues #227 and #218). I think it happens when units/batches are having zero spikes. In my case, it prompts an error here:
https://github.com/MouseLand/Kilosort2/blob/bdf9366bc73b4d9c101595c10193a532e33b54eb/mainLoop/learnTemplates.m#L149

>>> dbstop if error
>> master_kilosort
Looking for data inside ./ 
Time   0s. Computing whitening matrix.. 
Getting channel whitening matrix... 
Channel-whitening matrix computed. 
Time   1s. Loading raw data and applying filters... 
Time  34s. Finished preprocessing 801 batches. 
random seed for clusterSingleBatches: 1
Time  36s. Optimizing templates ...
One or more output arguments not assigned during call to "mexMPnu8".

Error in learnTemplates (line 149)
        mexMPnu8(Params, dataRAW, single(U), single(W), single(mu), iC-1, iW-1, UtU, iList-1, ...

Error in learnAndSolve8b (line 15)
    rez     = learnTemplates(rez, iorder0);

Error in master_kilosort (line 55)
rez = learnAndSolve8b(rez);
 
149         mexMPnu8(Params, dataRAW, single(U), single(W), single(mu), iC-1, iW-1, UtU, iList-1, ...

my Nfilt and W variables have the same values as yours, but the variable dWU returns a zero in the third dimension, which may be the origin of the problem (see #218):

K>> size(dWU)

ans =

    61    62     0

@mspacek
Copy link
Contributor Author

mspacek commented Sep 24, 2020

I worked around this by no longer exporting raw data with longish periods of zero-padding between recordings. I simply directly concatenate recordings from the same session, then later work out the absolute spike times relative to the start of the first recording of the session.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants