Is there a way to sort by group using only one sorter?

[mfvd]

Hello,

According to the documentation, it is possible to split by group and then use run_sorters to get a sorter object per group. However, if possible, I would like to a single object with data from all my groups (shanks)

In version 0.11.0 one could use grouping_property='group' to sort by group and to keep the data in a single sorter object. Is there a similar method with 0.93.0?

Cheers

[alejoe91]

HI @mfvd

Sure! Checkout the run_sorter_by_property() function! I think it's exactly what you need :)

[mfvd]

Hi @alejoe91 ,

Thanks!
I completely missed that one when reading the docs. I'll give it a try!

Cheers

[mfvd]

I've tried it out and notice that the shank number is equal for all channels. Also, when not specifying number of channels when exporting to phy, the channels in the WaveformView do not correspond to a single shank. Is my assumption that the sorter did not sort by shank correct?

If the data were sorted by shank I assume only the corresponding channels to be displayed for each unit and the shank number also to be the corresponding one.

# run sorter
sorted_ms4 = ss.run_sorter_by_property(sorter_name = 'mountainsort4',
                                       recording = rec_cashed,
                                       working_folder = 'ms4_output',  # 'sort_'+str(session)
                                       grouping_property = 'group',
                                       verbose = False,
                                       **default_ms4_params)

[alejoe91]

Ok that's a hidden one: after sorting by property if you want to export to phy and also keep the property grouping you have to set the sparsity correctly with the argument sparsity_dict=dict(method="by_property", by_property="group")

You can check the export_to_phy docs for more info ;)

[mfvd]

When checking the channel groups on the recording object I can see the group labels.
However, I get the following error:

AssertionError                            Traceback (most recent call last)
Input In [17], in <cell line: 19>()
     15 waveforms_ms4.run_extract_waveforms(n_jobs = -1, 
     16                                     chunk_size = 30000)
     18 # export spike to phy
---> 19 export_to_phy(waveforms_ms4, 
     20               './phy_ms4',
     21               sparsity_dict=dict(method="by_property", by_property="group"),
     22               compute_pc_features = True, 
     23               compute_amplitudes = True,
     24               progress_bar = True)
     26 print('Elapsed time: ', time.time()-t_start)

File /opt/anaconda3/envs/si_openephys/lib/python3.8/site-packages/spikeinterface/exporters/to_phy.py:143, in export_to_phy(waveform_extractor, output_folder, compute_pc_features, compute_amplitudes, sparsity_dict, copy_binary, max_channels_per_template, remove_if_exists, peak_sign, template_mode, dtype, verbose, **job_kwargs)
    140 templates = []
    141 templates_ind = []
--> 143 template_sparsity = get_template_channel_sparsity(waveform_extractor,
    144                                                   outputs="id", **sparsity_dict)
    145 num_sparse_chans = np.max([len(channels) for channels in template_sparsity.values()])
    146 num_channels = np.min([max_channels_per_template, num_sparse_chans])

File /opt/anaconda3/envs/si_openephys/lib/python3.8/site-packages/spikeinterface/toolkit/postprocessing/template_tools.py:172, in get_template_channel_sparsity(waveform_extractor, method, peak_sign, outputs, num_channels, radius_um, threshold, by_property)
    169 unit_index = waveform_extractor.sorting.id_to_index(unit_id)
    170 unit_property = waveform_extractor.sorting.get_property(by_property)[unit_index]
--> 172 assert unit_property in rec_by.keys(), f"Unit property {unit_property} cannot be found in the " \
    173                                        f"recording properties"
    174 chan_inds = waveform_extractor.recording.ids_to_indices(rec_by[unit_property].get_channel_ids())
    175 sparsity_with_index[unit_id] = chan_inds

AssertionError: Unit property 0 cannot be found in the recording properties

[alejoe91]

It seems that the recording doesn't have the group property from the message...can you check?

[mfvd]

Sorry for the late reply.

When I check with rec_cashed.get_property_keys(), the recording has a group property and the channels have assigned groups with rec_cashed.get_channel_groups().

I also check the sorting output and the units have assigned groups.
sorted_ms4.get_unit_property(12, 'group')

[alejoe91]

@mfvd what is the output of:

rec_by = rec_cashed.split_by("group")
print(rec_by.keys())

?

[mfvd]

Hi @alejoe91

The output is: dict_keys([0, 1, 2]).

[alejoe91]

what about np.unique(waveforms_ms4.sorting.get_property("group"))?

[mfvd]

I get the following: ['0' '1' '2']

[alejoe91]

Ah I see! There seems to be a mismatch in the dtype: groups for recordings are int anf for sorting are str..I'll look into it!

[alejoe91]

@mfvd should be fixed with this PR: #579

Can you try?