Post split crash

[catubc]

Hi. I am trying to sort a dataset with high rate of drift (made by Nick Steinmetz).

I am getting a crash quite late in the sorting process, it's unclear whether I used incorrect parameters, or this is a bug.

I am using an older run script which calls this function, maybe that's the issue:

% decide on cutoff
rez = set_cutoff(rez)

Thanks.

>> master_kilosort
Looking for data inside /media/cat/4TBSSD/nick_steinmetz_drift/rec1/ 
Time   0s. Computing whitening matrix.. 
Getting channel whitening matrix... 
Channel-whitening matrix computed. 
Time   4s. Loading raw data and applying filters... 
Time 218s. Finished preprocessing 966 batches. 
Obtained 7 PC waveforms in 1.40 seconds 
time 0.27, pre clustered 1 / 966 batches 
time 120.11, pre clustered 501 / 966 batches 
time 0.09, compared 1 / 966 batches 
time 42.16, compared 501 / 966 batches 
time 83.65, Re-ordered 966 batches. 
Time  89s. Optimizing templates ...
89.20 sec, 1 / 966 batches, 59 units, nspks: 4.4537, mu: 17.2255, nst0: 39, merges: 0.0000, 0.0000 
108.27 sec, 101 / 966 batches, 470 units, nspks: 1282.2824, mu: 16.4949, nst0: 1500, merges: 95.4677, 0.1260 
130.99 sec, 201 / 966 batches, 509 units, nspks: 958.8761, mu: 16.2488, nst0: 1308, merges: 116.3460, 0.1239 
153.80 sec, 301 / 966 batches, 501 units, nspks: 1085.8357, mu: 15.4025, nst0: 1124, merges: 114.0251, 0.0464 
177.05 sec, 401 / 966 batches, 615 units, nspks: 934.4217, mu: 15.3302, nst0: 723, merges: 120.0968, 0.0056 
201.42 sec, 501 / 966 batches, 631 units, nspks: 1004.6619, mu: 15.2641, nst0: 972, merges: 121.4152, 0.0007 
226.20 sec, 601 / 966 batches, 659 units, nspks: 915.2707, mu: 15.1995, nst0: 597, merges: 117.8756, 0.0001 
250.41 sec, 701 / 966 batches, 610 units, nspks: 942.5358, mu: 14.8994, nst0: 1021, merges: 120.6000, 0.0000 
273.45 sec, 801 / 966 batches, 569 units, nspks: 925.5024, mu: 14.9946, nst0: 1023, merges: 111.4591, 0.0349 
295.25 sec, 901 / 966 batches, 548 units, nspks: 837.2383, mu: 15.1695, nst0: 712, merges: 110.5325, 0.0698 
Elapsed time is 310.020544 seconds.
memorized middle timepoint 
Finished learning templates 
Time 313s. Optimizing templates ...
313.43 sec, 1 / 483 batches, 528 units, nspks: 218.5000, mu: 14.2196, nst0: 23326 
384.94 sec, 101 / 483 batches, 528 units, nspks: 1615.8264, mu: 13.7983, nst0: 21778 
465.08 sec, 201 / 483 batches, 528 units, nspks: 1710.9239, mu: 13.4440, nst0: 23645 
548.76 sec, 301 / 483 batches, 528 units, nspks: 1796.7308, mu: 13.1407, nst0: 25018 
633.43 sec, 401 / 483 batches, 528 units, nspks: 1635.1845, mu: 12.8721, nst0: 23917 
Elapsed time is 708.383237 seconds.
Time 714s. Optimizing templates ...
715.36 sec, 1 / 483 batches, 528 units, nspks: 233.9500, mu: 14.2196, nst0: 23686 
782.18 sec, 101 / 483 batches, 528 units, nspks: 1356.3754, mu: 13.7875, nst0: 20101 
857.61 sec, 201 / 483 batches, 528 units, nspks: 1927.2167, mu: 13.3633, nst0: 23333 
937.57 sec, 301 / 483 batches, 528 units, nspks: 1658.1140, mu: 12.8825, nst0: 23720 
1015.69 sec, 401 / 483 batches, 528 units, nspks: 1856.8295, mu: 12.6838, nst0: 22227 
Elapsed time is 1086.824713 seconds.

ans =

    21763708

Finished compressing time-varying templates 
initialized spike counts
merged 362 into 361 
merged 398 into 390 
merged 130 into 131 
merged 493 into 500 
merged 412 into 413 
merged 464 into 466 
merged 475 into 471 
merged 364 into 368 
merged 115 into 121 
merged 63 into 65 
merged 525 into 528 
merged 168 into 158 
merged 390 into 391 
merged 379 into 387 
Found 0 splits, checked 1/528 clusters, nccg 0 
Found 1 splits, checked 1/529 clusters, nccg 0 
Found 2 splits, checked 1/530 clusters, nccg 0 
Found 3 splits, checked 1/531 clusters, nccg 0 
Found 4 splits, checked 1/532 clusters, nccg 0 
Found 5 splits, checked 1/533 clusters, nccg 0 
Found 79 splits, checked 101/607 clusters, nccg 66 
Found 80 splits, checked 101/608 clusters, nccg 66 
Found 81 splits, checked 101/609 clusters, nccg 66 
Found 82 splits, checked 101/610 clusters, nccg 66 
Found 83 splits, checked 101/611 clusters, nccg 66 
Found 147 splits, checked 201/675 clusters, nccg 137 
Found 148 splits, checked 201/676 clusters, nccg 137 
Found 209 splits, checked 301/737 clusters, nccg 207 
Found 260 splits, checked 401/788 clusters, nccg 283 
Found 261 splits, checked 401/789 clusters, nccg 283 
Found 262 splits, checked 401/790 clusters, nccg 283 
Found 263 splits, checked 401/791 clusters, nccg 283 
Found 322 splits, checked 501/850 clusters, nccg 355 
Found 323 splits, checked 501/851 clusters, nccg 355 
Found 395 splits, checked 601/923 clusters, nccg 429 
Found 396 splits, checked 601/924 clusters, nccg 429 
Found 455 splits, checked 701/983 clusters, nccg 500 
Found 540 splits, checked 801/1068 clusters, nccg 561 
Found 541 splits, checked 801/1069 clusters, nccg 561 
Found 609 splits, checked 901/1137 clusters, nccg 630 
Found 670 splits, checked 1001/1198 clusters, nccg 677 
Found 671 splits, checked 1001/1199 clusters, nccg 677 
Found 751 splits, checked 1101/1279 clusters, nccg 727 
Found 814 splits, checked 1201/1342 clusters, nccg 774 
Found 815 splits, checked 1201/1343 clusters, nccg 774 
Found 816 splits, checked 1201/1344 clusters, nccg 774 
Found 877 splits, checked 1301/1405 clusters, nccg 813 
Found 878 splits, checked 1301/1406 clusters, nccg 813 
Found 939 splits, checked 1401/1467 clusters, nccg 848 
Found 982 splits, checked 1501/1510 clusters, nccg 890 
Finished splitting. Found 987 splits, checked 1515/1515 clusters, nccg 895 
Found 0 splits, checked 1/1515 clusters, nccg 0 
Found 19 splits, checked 101/1534 clusters, nccg 62 
Found 38 splits, checked 201/1553 clusters, nccg 133 
Found 59 splits, checked 301/1574 clusters, nccg 200 
Found 80 splits, checked 401/1595 clusters, nccg 272 
Found 91 splits, checked 501/1606 clusters, nccg 341 
Found 112 splits, checked 601/1627 clusters, nccg 407 
Found 113 splits, checked 601/1628 clusters, nccg 407 
Found 114 splits, checked 601/1629 clusters, nccg 407 
Found 119 splits, checked 701/1634 clusters, nccg 475 
Found 146 splits, checked 801/1661 clusters, nccg 527 
Found 169 splits, checked 901/1684 clusters, nccg 591 
Found 180 splits, checked 1001/1695 clusters, nccg 633 
Found 203 splits, checked 1101/1718 clusters, nccg 676 
Found 222 splits, checked 1201/1737 clusters, nccg 719 
Found 235 splits, checked 1301/1750 clusters, nccg 750 
Found 236 splits, checked 1301/1751 clusters, nccg 750 
Found 247 splits, checked 1401/1762 clusters, nccg 788 
Found 254 splits, checked 1501/1769 clusters, nccg 822 
Found 273 splits, checked 1601/1788 clusters, nccg 887 
Found 306 splits, checked 1701/1821 clusters, nccg 928 
Found 327 splits, checked 1801/1842 clusters, nccg 970 
Finished splitting. Found 332 splits, checked 1847/1847 clusters, nccg 997 
Error using  / 
Matrix dimensions must agree.

Error in ccg (line 55)
Q00 = sum(K(irange1)) / (numel(irange1)*tbin* numel(st1) * numel(st2)/T);

Error in set_cutoff (line 56)
    [K, Qi, Q00, Q01, rir] = ccg(st, st, 500, dt); % % compute the auto-correlogram with 500 bins at 1ms bins

Error in master_kilosort (line 51)
rez = set_cutoff(rez);

[marius10p]

Something strange seems to have happened. There are too many spikes and way too many splits. Can you please check in the GUI that the channel configuration is correct?

[catubc]

Yes, that's quite possible that I'm using 2Col vs. 4Col layout. I will re-run and update. Thanks!

[catubc]

Yes, KS2 ran fully with updated 385 channel map. Thanks!

[FilippoHei]

I have a very similar problem at the moment. I checked my channel map and it is all fine. I also sorted several rhd files just like this one (also with the same channel map), and never ran into troubles. Any suggestions?

There do seem to be many "duplicate spike pairs", but I didn't struggle with these in previous recordings.

Time   0s. Determining good channels.. 
found 479712 threshold crossings in 183.68 seconds of data 
found 0 bad channels 
Time  18s. Computing whitening matrix.. 
Getting channel whitening matrix... 
Channel-whitening matrix computed. 
Time  20s. Loading raw data and applying filters... 
Time  90s. Finished preprocessing 420 batches. 
random seed for clusterSingleBatches: 1
Obtained 7 PC waveforms in 0.77 seconds 
time 0.07, pre clustered 1 / 420 batches 
time 0.01, compared 1 / 420 batches 
time 4.08, Re-ordered 420 batches. 
Time   5s. Optimizing templates ...
useStableMode selected but STABLEMODE not enabled in compiled mexMPnu8.
4.97 sec, 1 / 420 batches, 13 units, nspks: 1.5510, mu: 15.7756, nst0: 20, merges: 0.0000, 0.0000 
10.20 sec, 101 / 420 batches, 115 units, nspks: 839.0870, mu: 13.9513, nst0: 1505, merges: 27.9846, 0.0000 
16.64 sec, 201 / 420 batches, 112 units, nspks: 667.5174, mu: 10.8574, nst0: 497, merges: 34.8523, 0.0000 
23.17 sec, 301 / 420 batches, 97 units, nspks: 1285.8475, mu: 10.0000, nst0: 825, merges: 35.0784, 0.0000 
30.58 sec, 401 / 420 batches, 155 units, nspks: 1104.4121, mu: 13.9263, nst0: 552, merges: 33.0036, 0.0000 
Elapsed time is 32.112720 seconds.
Finished learning templates 
Time  32s. Optimizing templates ...
32.56 sec, 1 / 210 batches, 162 units, nspks: 54.7000, mu: 14.2170, nst0: 6826 
56.19 sec, 101 / 210 batches, 162 units, nspks: 1454.8713, mu: 13.5679, nst0: 10291 
80.07 sec, 201 / 210 batches, 162 units, nspks: 1055.4020, mu: 12.9027, nst0: 7406 
Elapsed time is 82.204233 seconds.
Time  83s. Optimizing templates ...
82.90 sec, 1 / 210 batches, 162 units, nspks: 63.9000, mu: 14.2170, nst0: 7926 
102.71 sec, 101 / 210 batches, 162 units, nspks: 1353.6871, mu: 13.7276, nst0: 5371 
122.95 sec, 201 / 210 batches, 162 units, nspks: 1385.8472, mu: 13.0035, nst0: 8076 
Elapsed time is 124.796119 seconds.
Number of spikes before applying cutoff: 3110764
Finished compressing time-varying templates 
Now comparing spikes separated by 0 other spikes.
On iteration 1, 682511 duplicate spike pairs were identified.
Now comparing spikes separated by 1 other spikes.
On iteration 2, 471278 duplicate spike pairs were identified.
Now comparing spikes separated by 2 other spikes.
On iteration 3, 268695 duplicate spike pairs were identified.
Now comparing spikes separated by 3 other spikes.
On iteration 4, 134802 duplicate spike pairs were identified.
Now comparing spikes separated by 4 other spikes.
On iteration 5, 62055 duplicate spike pairs were identified.
Now comparing spikes separated by 5 other spikes.
On iteration 6, 26867 duplicate spike pairs were identified.
Now comparing spikes separated by 6 other spikes.
On iteration 7, 11073 duplicate spike pairs were identified.
Now comparing spikes separated by 7 other spikes.
On iteration 8, 4463 duplicate spike pairs were identified.
Now comparing spikes separated by 8 other spikes.
On iteration 9, 1874 duplicate spike pairs were identified.
Now comparing spikes separated by 9 other spikes.
On iteration 10, 751 duplicate spike pairs were identified.
Now comparing spikes separated by 10 other spikes.
On iteration 11, 331 duplicate spike pairs were identified.
Now comparing spikes separated by 11 other spikes.
On iteration 12, 151 duplicate spike pairs were identified.
Now comparing spikes separated by 12 other spikes.
On iteration 13, 87 duplicate spike pairs were identified.
Now comparing spikes separated by 13 other spikes.
On iteration 14, 55 duplicate spike pairs were identified.
Now comparing spikes separated by 14 other spikes.
On iteration 15, 32 duplicate spike pairs were identified.
Now comparing spikes separated by 15 other spikes.
On iteration 16, 30 duplicate spike pairs were identified.
Now comparing spikes separated by 16 other spikes.
On iteration 17, 16 duplicate spike pairs were identified.
Now comparing spikes separated by 17 other spikes.
On iteration 18, 22 duplicate spike pairs were identified.
Now comparing spikes separated by 18 other spikes.
On iteration 19, 22 duplicate spike pairs were identified.
Now comparing spikes separated by 19 other spikes.
On iteration 20, 22 duplicate spike pairs were identified.
Now comparing spikes separated by 20 other spikes.
On iteration 21, 11 duplicate spike pairs were identified.
Now comparing spikes separated by 21 other spikes.
On iteration 22, 12 duplicate spike pairs were identified.
Now comparing spikes separated by 22 other spikes.
On iteration 23, 9 duplicate spike pairs were identified.
Now comparing spikes separated by 23 other spikes.
On iteration 24, 9 duplicate spike pairs were identified.
Now comparing spikes separated by 24 other spikes.
On iteration 25, 4 duplicate spike pairs were identified.
Now comparing spikes separated by 25 other spikes.
On iteration 26, 6 duplicate spike pairs were identified.
Now comparing spikes separated by 26 other spikes.
On iteration 27, 6 duplicate spike pairs were identified.
Now comparing spikes separated by 27 other spikes.
On iteration 28, 3 duplicate spike pairs were identified.
Now comparing spikes separated by 28 other spikes.
On iteration 29, 4 duplicate spike pairs were identified.
Now comparing spikes separated by 29 other spikes.
On iteration 30, 3 duplicate spike pairs were identified.
Now comparing spikes separated by 30 other spikes.
Now comparing spikes separated by 31 other spikes.
Now comparing spikes separated by 32 other spikes.
Now comparing spikes separated by 33 other spikes.
No remaining simultaneous spikes.
Removing 1.10295e+06 spikes from rez structure.
initialized spike counts
Found 1 splits, checked 1/163 clusters, nccg 0 
Found 7 splits, checked 101/169 clusters, nccg 0 
Finished splitting. Found 20 splits, checked 182/182 clusters, nccg 0 
Found 0 splits, checked 1/182 clusters, nccg 0 
Found 1 splits, checked 101/183 clusters, nccg 0 
Finished splitting. Found 3 splits, checked 185/185 clusters, nccg 0 

Error using  / 
Matrix dimensions must agree.
Error in ccg (line 55)
Q00 = sum(K(irange1)) / (numel(irange1)*tbin* numel(st1) * numel(st2)/T);
Error in set_cutoff (line 58)
    [K, Qi, Q00, Q01, rir] = ccg(st, st, 500, dt); % % compute the auto-correlogram with 500 bins at 1ms bins
Error in main_kilosort (line 172)
rez = set_cutoff(rez);