-
Notifications
You must be signed in to change notification settings - Fork 230
Implement ChunkExecutor
#4216
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Implement ChunkExecutor
#4216
Conversation
|
|
||
|
|
||
| def get_random_recording_slices( | ||
| def get_random_slices( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is too neutral no ? Lets find some thing like
get_random_slices_along_time
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
get_random_time_slices?
|
Lets discuss this in detail. This is lots some semantic changes. |
What about a
|
This PR makes a
ChunkExecutorwhich is agnostic to the recording and applies to any chunkableBaseExtractor(which definesget_sample_size(), get_num_samples(), get_num_segments()This is because we are working on a new "spikeinterface" for calcium imaging, and this refactor makes it suuuuper straightforward to use the same parallelization machinery for
Imagingobjects (which inherit from theBaseExtractor)Update 29/11/25
Following up on trying to make SI tools as general as possible for its calcium imaging "clone" (photon-mosaic) I made some more dramatic refactoring:
ChunkableMixinabstract mixin that implements all method needed by theChunkExecutorBaseRecordingspecific methods (e.g.,write_binary/write_memoryand more) are now in thechunkable_toolsand applicable to anyChunkableMixinobjectChunkableMixinhas a generalget_datafunction, so it can be used by many derived classes (e.g.,get_tracesfor recordings,get_videos/seriesfor imaging)get_shape(segment_index), which returns the buffer shape by segment_recording_segmentsand sorting objects_sorting_segments. Each class then implemented its ownadd_*function. Since segments are aBaseExtractorconcept, I centralized the definition (self.segments: List[BaseSegment] = []), added aself.segmentsproperty, and a centralizedadd_segmentfunction.@samuelgarcia @chrishalcrow @yger @h-mayorquin @JoeZiminski @zm711
Let me know what you think!!!