-
Notifications
You must be signed in to change notification settings - Fork 175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add new dataset class using zarr
file storage.
#478
Conversation
self, | ||
train_indices=None, | ||
test_indices=None, | ||
): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you add docstring?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Co-authored-by: Sylvain Chevallier <sylvain.chevallier@universite-paris-saclay.fr>
Co-authored-by: Sylvain Chevallier <sylvain.chevallier@universite-paris-saclay.fr>
Co-authored-by: Sylvain Chevallier <sylvain.chevallier@universite-paris-saclay.fr>
Co-authored-by: Sylvain Chevallier <sylvain.chevallier@universite-paris-saclay.fr>
Co-authored-by: Sylvain Chevallier <sylvain.chevallier@universite-paris-saclay.fr>
Co-authored-by: Sylvain Chevallier <sylvain.chevallier@universite-paris-saclay.fr>
Co-authored-by: Sylvain Chevallier <sylvain.chevallier@universite-paris-saclay.fr>
Codecov Report
@@ Coverage Diff @@
## master #478 +/- ##
==========================================
- Coverage 84.55% 83.46% -1.10%
==========================================
Files 63 63
Lines 4676 4760 +84
==========================================
+ Hits 3954 3973 +19
- Misses 722 787 +65 |
As we found current solution to be performant enough, closing this for now. |
Pull Request Description
This PR introduces a new dataset implementation aimed at improving dataset loading speed. Additionally, it includes a BCI decoding example in a Colab notebook for reference: Link to Colab Example.
Notes
Considered Alternatives: During the development process, we explored potential alternatives like numpy.memmap and lmdb. However, we encountered limitations with numpy.memmap, particularly in terms of lacking compression options. LMDB, while efficient, isn't well-suited for lazy loading, useful for querying windows from recordings.
HDF5 vs. Zarr Debate: There's an ongoing debate between HDF5 and Zarr. Both have their respective advantages, but Zarr appears to have more active development. We might also allow users to choose between both options.