fix(datasets): harden dataset loading for large pretrain mixtures#304

Merged

shuheng-liu merged 18 commits into

mainfrom

claude/intelligent-chatelet-f16fb0

May 14, 2026

Commits on May 14, 2026

fix(datasets): guard get_safe_version with is_valid_version on SHA revisions
shuheng-liu
committed
fix(datasets): fall back to inferred features when info.json cast fails
shuheng-liu
committed
fix(datasets): use HF Hub filelock instead of rank-coordinated metadata download
shuheng-liu
committed
fix(datasets): always barrier in load_or_compute_speed_percentiles
shuheng-liu
committed
build(deps): add py-spy to dev extras for deadlock debugging
shuheng-liu
committed
Revert "build(deps): add py-spy to dev extras for deadlock debugging"
shuheng-liu
committed
Revert "fix(datasets): use HF Hub filelock instead of rank-coordinated metadata download"
shuheng-liu
committed
fix(datasets): skip episodes with unresolvable task labels in speed bucketing
shuheng-liu
committed
fix(datasets): also fall back on ValueError Keys mismatch, not just TypeError
shuheng-liu
committed
fix(datasets): fall back on any typed-Dataset schema error, truncate log
shuheng-liu
committed
fix(datasets): stream parquet to mmap'd Arrow file instead of full in-RAM load
shuheng-liu
committed
fix(datasets): write Arrow IPC stream format for Dataset.from_file mmap
shuheng-liu
committed
fix(datasets): use load_dataset(parquet) for genuine mmap, drop hand-rolled paths
shuheng-liu
committed
fix(datasets): download episode files directly, skip slow snapshot_download filter
shuheng-liu
committed
fix(datasets): route whole-repo pulls to snapshot_download, skip on-disk files
shuheng-liu
committed
test(datasets): mock hf_hub_download in lerobot_dataset_factory
shuheng-liu
committed
refactor(datasets): address #304 review feedback
shuheng-liu
committed
docs(datasets): clarify load_hf_dataset schema-inference docstring
shuheng-liu
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(datasets): harden dataset loading for large pretrain mixtures#304

fix(datasets): harden dataset loading for large pretrain mixtures#304
shuheng-liu merged 18 commits into
mainfrom
claude/intelligent-chatelet-f16fb0

Commits on May 14, 2026

fix(datasets): guard get_safe_version with is_valid_version on SHA revisions

fix(datasets): fall back to inferred features when info.json cast fails

fix(datasets): use HF Hub filelock instead of rank-coordinated metadata download

fix(datasets): always barrier in load_or_compute_speed_percentiles

build(deps): add py-spy to dev extras for deadlock debugging

Revert "build(deps): add py-spy to dev extras for deadlock debugging"

Revert "fix(datasets): use HF Hub filelock instead of rank-coordinated metadata download"

fix(datasets): skip episodes with unresolvable task labels in speed bucketing

fix(datasets): also fall back on ValueError Keys mismatch, not just TypeError

fix(datasets): fall back on any typed-Dataset schema error, truncate log

fix(datasets): stream parquet to mmap'd Arrow file instead of full in-RAM load

fix(datasets): write Arrow IPC stream format for Dataset.from_file mmap

fix(datasets): use load_dataset(parquet) for genuine mmap, drop hand-rolled paths

fix(datasets): download episode files directly, skip slow snapshot_download filter

fix(datasets): route whole-repo pulls to snapshot_download, skip on-disk files

test(datasets): mock hf_hub_download in lerobot_dataset_factory

refactor(datasets): address #304 review feedback

docs(datasets): clarify load_hf_dataset schema-inference docstring