v0.10.1

A bug-fix release. Several blocks that crashed on empty text or on numpy and pandas values now handle them. HuggingFaceDatasetReader now applies skip. The test suite now includes Hypothesis property tests, and CI is faster.

Behaviour changes

These fixes can change output for runs that used to crash or ignore a parameter. Runs that worked before give the same results.

• Stats: DocStats, LineStats, ParagraphStats, SentenceStats, WordStats and WordsContaminationStats no longer raise ZeroDivisionError on empty, whitespace-only or punctuation-only documents. Their ratios and averages are recorded as 0.0 and included in summaries. Use count stats such as length or n_words to find these documents (#512).
• GopherRepetitionFilter: empty documents no longer raise ZeroDivisionError (#528).
• HuggingFaceDatasetReader: skip was documented but ignored. It now skips the first skip documents with text in each task. Generated ids do not depend on skip (#544).
• limit and skip are per task on all readers. This was already the behaviour and is now documented: with N tasks, you get up to N × limit documents in total. Rank 0 logs the total (#544).
• JsonlWriter: metadata with numpy scalars (bool, int, float16/32/64, datetime64) or pandas.Timestamp is now written instead of raising TypeError. datetime64 keeps its full precision, NaT is written as null, and NaN/inf are null as for Python floats. Arrays, timedelta64, complex and longdouble still raise (#490, #543).

What's Changed

• docs(jobs): drop the pre-release pin now 0.10.0 is out by @davanstrien in #511
• chore: enable Dependabot weekly GitHub Actions bumps by @hf-dependantbot-rollout[bot] in #487
• Bump the actions group with 5 updates by @dependabot[bot] in #525
• fix(filters): GopherRepetitionFilter raises ZeroDivisionError on empty documents by @Linxiushen in #528
• Fix JsonlWriter crash on pandas.Timestamp metadata by @discobot in #490
• test: add Hypothesis property tests for filters, stats and JSONL by @davanstrien in #534
• Enable ruff flake8-bugbear (B) rules by @davanstrien in #535
• fix(writers): clear operations after commit in HuggingFaceDatasetWriter by @apmoore1 in #502
• chore(deps): bump trufflesecurity/trufflehog from 3.97.4 to 3.97.6 in the actions group by @dependabot[bot] in #539
• ci: run tests in parallel with pytest-xdist by @davanstrien in #536
• ci: install tests without the GPU stack by @davanstrien in #537
• fix: import importlib.metadata and importlib.util explicitly by @davanstrien in #540
• test: property tests for sharded reads, Parquet round trip and exact dedup by @davanstrien in #542
• fix(stats): handle empty documents by @Excelius-Wang in #512
• fix(writers): write numpy scalars in JsonlWriter metadata by @davanstrien in #543
• fix(readers): apply skip in HuggingFaceDatasetReader; document limit/skip as per task by @davanstrien in #544
• release: bump version to 0.10.1 by @davanstrien in #546

New Contributors

• @hf-dependantbot-rollout[bot] made their first contribution in #487
• @Linxiushen made their first contribution in #528
• @discobot made their first contribution in #490
• @apmoore1 made their first contribution in #502
• @Excelius-Wang made their first contribution in #512

Full Changelog: v0.10.0...v0.10.1