Add Hugging Face Hub integration with Zarr format for dataset sharing

[Kkuntal990]

Summary

Adds Hugging Face Hub integration to braindecode, enabling easy sharing, versioning, and collaboration on EEG datasets. Datasets can be uploaded to and downloaded from the Hub using optimized Zarr format.

Supports three dataset types:

• WindowsDataset - Epoched data (mne.Epochs-based)
• EEGWindowsDataset - Continuous raw data with windowing metadata
• RawDataset - Continuous raw data without windowing

Key Features

Hub Integration:

• push_to_hub() - Upload datasets with one command
• from_pretrained() - Download shared datasets
• Automatic README generation with metadata and usage info
• Private/public repositories with version control

Zarr Format (Optimized for Training):

• Fastest random access: 0.010 ms (critical for PyTorch DataLoader)
• ~23% size reduction with blosc compression
• Lossless float32 precision storage
• Explicit dtype specification for clarity
• Preserves all MNE metadata, channel info, and preprocessing history

Note: Hub uses Zarr exclusively. Braindecode already supports local .fif format via dataset.save() and load_concat_dataset().

Architecture

Registry Pattern - Eliminates circular imports by using a central registry where dataset classes register themselves. Both hub.py and base.py can access the registry without importing each other.

Validation System:

• Enforces dataset type consistency (no mixing types)
• Checks channel names and sampling frequency uniformity

Integration with PR #813:

• Compatible with improved preprocessor serialization
• Preprocessing kwargs properly initialized on all dataset types

Changes

Modified Files:

• braindecode/datasets/base.py - Registry decorators, preprocessing kwargs init
• braindecode/datasets/hub.py - Core Hub logic, validation, dataset card
• braindecode/datautil/hub_formats.py - Delegation pattern (67% duplication eliminated)
• braindecode/datasets/registry.py - Registry pattern (NEW)
• examples/datasets_io/plot_hub_integration.py - Comprehensive example

Tests:

• test/unit_tests/datasets/test_hub_integration.py - 19 tests
• test/unit_tests/datautil/test_hub_formats.py - 9 tests
• Total: 28 tests, all passing ✅, 0 warnings

Test Coverage: Hub methods, zarr round-trip (all 3 types), validation, registry, overwrite

Live Demo Datasets

Successfully uploaded and tested (post upstream/master merge):

• https://huggingface.co/datasets/Kkuntal990/bnci-windows
• https://huggingface.co/datasets/Kkuntal990/bnci-eegwindows
• https://huggingface.co/datasets/Kkuntal990/bnci-raw

All verified for download, data integrity, and PyTorch DataLoader compatibility.

Checklist

• All tests passing (28/28, 0 warnings)
• Documentation updated (examples, docstrings)
• No breaking changes to existing API
• Error handling with helpful messages
• Registry pattern eliminates circular imports
• All three dataset types supported
• Float32 precision with explicit dtype
• Compatible with PR Improve Preprocessor serialisation  #813 (preprocessor serialization)
• Production ready (live datasets tested post-merge)

[bruAristimunha]

@PierreGtch , can you take a quick look please?

[bruAristimunha]

Nice first round

[PierreGtch]

Another nice PR @Kkuntal990

a few small comments, but the main two are:

• You need to take into account the three types of RecordDataset we have, or raise a clear no implemented error. Currently only two are considered
• different dataset in the same concat can have different information. Two options:
  • 1. you chech that all the elements in the concat have the EXACT info, and raise an error if not. This will be easier to browse datasets through the hub
  • 2. save the different info of each dataset contained in the concat separately

I'm more in favour of 1.

[Kkuntal990]

Another nice PR @Kkuntal990

a few small comments, but the main two are:

• You need to take into account the three types of RecordDataset we have, or raise a clear no implemented error. Currently only two are considered

• different dataset in the same concat can have different information. Two options:

  • 1. you chech that all the elements in the concat have the EXACT info, and raise an error if not. This will be easier to browse datasets through the hub
  • 2. save the different info of each dataset contained in the concat separately

I'm more in favour of 1.

• Handled all 3 datasets
• Raising error if concatenation has different datasets. Added a test for that error. Is it fine for now or do we need to handle mixed datasets, it will be a little tricky.

[Kkuntal990]

Hi @PierreGtch @bruAristimunha , I have made some architectural changes to the PR and left couple of minor comments open as I need more clarification on those.

Also added some info about the features and comparison with the hugging face, in case it is helpful.

Thank you for your time!

Also sorry for the big PR. Next time I will modularize it better

[Copilot]

Pull Request Overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Similar RST syntax issue: mixing :: with explicit .. code-block:: directive. The :: at the end of line 932 should be removed, or the entire block should use simple literal block syntax (just :: with indentation).

Recommended fix - use markdown code fences for consistency with the rest of the README:

## Using with PyTorch DataLoader

```python
from torch.utils.data import DataLoader
...

```suggestion
```python
from torch.utils.data import DataLoader

# Create DataLoader for training
train_loader = DataLoader(
    dataset,
    batch_size=32,
    shuffle=True,
    num_workers=4
)

# Training loop
for X, y, metainfo in train_loader:
    # X shape: [batch_size, n_channels, n_times]
    # y shape: [batch_size]
    # metainfo shape: [batch_size, 2] (start and end indices)
    # Process your batch...

[PierreGtch]

@copilot open a new pull request to apply changes based on this feedback

[bruAristimunha]

okay 105 msgs in this PR, it is too much ;p

[bruAristimunha]

many thanks for the all the work @Kkuntal990