WaveformExtractor.run_extract_waveforms has to run on all units

[DradeAW]

Hi,

I want to extract the waveforms of multiple units, my understanding is to use the WaveformExtractor, set the parameters, and run run_extract_waveforms().
Except I'm not seeing a parameter to give which unit ids I want to extract, which means I need to extract the waveforms of all the units (which seems like a potential waste of time, and a lot of writing to disk when using max_spikes_per_unit=None).

In my use case, it wrote 298.7 GB to the disk (taking a few minutes) when extracting all of the waveforms (96 units) when I actually was only interested in 3 of them ...

Is it made like this on purpose?
Or is it something that needs to be added?

Thanks,
Drade

[alejoe91]

Hi @Dradeliomecus

I think it totally makes sense to have an argument to only extract a subset of units. Would you have time to make a PR about that? It shouldn't take too much time :)

Let me know!
Alessio

[DradeAW]

I should be able to do it tomorrow morning!

[alejoe91]

I should be able to do it tomorrow morning!

Awesome!!! thanks :)

[samuelgarcia]

Or you create a sub sorting with sorting.select_units and run another waveform extractor on it.
sounds easier no ?

[DradeAW]

Or you create a sub sorting with sorting.select_units and run another waveform extractor on it. sounds easier no ?

It's easier if I only need a subset of units and that's all.
But often, I need to get the waveforms of a subset of units, and then depending on the result get the waveforms of a different subset of units.

Being able to run run_extract_waveforms multiple times of a different subset of units while always remembering the previous results could be a really nice feature I think.

[DradeAW]

Here is the pull request (I tested it on my dataset, and it works both with and without specifying unit_ids):
#332

[alejoe91]

@Dradeliomecus after some discussion we think that the best option is to pre-select units. In fact, if we allow to compute users on a subset of units then it would be hard to keep track of which units have been already computed or not.

Thanks for raising the issue though! Happy to discuss more but closing for now

[DradeAW]

Alright I understand!

I just thought that keeping track of units already computed could improve performance in certain use cases, but maybe it's too niche or maybe it's just too complicated as you say.