RawWaveforms does not match the number of clusters

[laurah1010]

When I run the extract_raw_data_demo I get less waveforms than expected. This then results in errors when I try to run UnitMatch as it looks for waveforms that do not exist. This happens for all datasets I try to run.

[EnnyvanBeest]

Hi,

could you give a bit more details? How many waveforms do you expect, how many do you get?
What recording probe, software, and preprocessing have you done prior to this?

[laurah1010]

Thank you for the quick response!
In one example I would expect to get 231 based on the number of clusters in my cluster_info.tsv file but I am only getting 223 waveforms extracted.
I am recording with Neuropixels 2.0 probe and SpikeGLX software and I preprocess with spikeinterface then Kilosort 3.
Also, as an extra note, I am running this with "extract_good_units_only = False". When I set it as True then no raw waveforms get extracted but no errors arise.

[EnnyvanBeest]

I pushed some changes to the python version recently that have to do with extraction of data if you used an older version from Kilosort, so it may be worth redownloading unitmatch from scratch.

In either case, you seem to be using cluster_info.tsv which is a file created by manual curation with phy(2), the pipeline uses cluster_group.tsv when good_units_only is true, and cluster_ids as found in spike_clusters.npy when extracting all neurons. My suspicion is that some of the clusters in cluster_info.tsv are empty after having merged two units into one. The old 'cluster_id' ghost remains, but it does not have spikes anymore and hence no extracted waveform to be stored for it (as it no longer appears in spike_clusters.npy).

The best strategy for your data would be to keep it extract_good_units_only = False, and then use cluster_info.tsv to define good_units for the rest of the pipeline (overwrite clus_info.good_units). My suspicion is that some of the clusters in cluster_info.tsv are empty after having merged two units into one. The old 'cluster_id' ghost remains, but it does not have spikes anymore and hence no extracted waveform to be stored for it.

[laurah1010]

I have discovered that the issues arises because the Unit # that is being assigned to the saved RawWaveforms does not match the cluster ids. The number of clusters is actually the same but since the cluster ids from kilosort are not sequential, the values for the two parameters do not align.

[EnnyvanBeest]

Apologies for the late reply, and thanks for letting us know. We're looking into fixing this/clarifying this. In the meantime, I hope you can use the waveforms now you know what the discrepancy is!

[EnnyvanBeest]

Hi Laura, can you confirm that new versions of UnitMatch now work for you? If not, can you please open an issue? Thanks!