v0.10.0

Adds JobsPipelineExecutor (experimental) for running pipelines on Hugging Face Jobs, and support for Hugging Face storage buckets as a datatrove DataFolder.

What's Changed

• Add AGENTS.md with development guidelines by @JoelNiklaus in #468
• Add tests for untested modules and integration patterns by @JoelNiklaus in #469
• Organize test suite into logical subdirectories by @JoelNiklaus in #470
• Fix check_dataset entry point by adding missing main function by @JoelNiklaus in #471
• Relax version upper bounds on huggingface-hub and transformers by @JoelNiklaus in #472
• Add datatrove tag to synthetic dataset cards by @JoelNiklaus in #473
• Add Python 3.13 support by @JoelNiklaus in #474
• 🔒 Pin GitHub Actions to commit SHAs by @paulinebm in #475
• Migrate to uv and bump vllm / transformers / hfh by @lewtun in #481
• Update uv.lock to bump vllm to 0.19.1 by @JoelNiklaus in #482
• Remove transformers>=5.0 uv overrides by @JoelNiklaus in #483
• Add HuggingFace storage bucket support by @JoelNiklaus in #484
• Add JobsPipelineExecutor: run datatrove pipelines on HF Jobs (experimental) by @davanstrien in #497
• fix(jobs): default python= to the coordinator's version by @davanstrien in #498
• Preserve chat reasoning outputs in inference results by @edbeeching in #496
• feat(jobs): env= passthrough on JobsPipelineExecutor by @davanstrien in #499
• docs: add ParquetWriter class docstring by @davanstrien in #503
• fix: raise huggingface-hub floor to 1.6.0 by @davanstrien in #504
• ci: switch PyPI release to trusted publishing (OIDC) by @davanstrien in #505
• release: bump version to 0.10.0rc1 by @davanstrien in #506
• fix: pin xxhash<4 (4.0.0 breaks all dedup pipelines) by @davanstrien in #507
• docs(jobs): point JobsPipelineExecutor install guidance at the released package by @davanstrien in #509
• release: bump version to 0.10.0 by @davanstrien in #510

New Contributors

• @paulinebm made their first contribution in #475
• @davanstrien made their first contribution in #497
• @edbeeching made their first contribution in #496

Full Changelog: v0.9.0...v0.10.0