Repository navigation
dataprof v0.13.1
0.13.0 shipped with a confession in its known limitations: some JSON bytes behaved differently from the same bytes in a file. Three days later that list is shorter. This is a patch release: nothing removed, one rule added, and the wheels rebuilt on patched zstd.
The 0.13 audits profiled the same bytes through every route and compared the answers. The routes that still disagreed were on the synchronous bytes path, which decodes JSON with Python's json module, while files, async reads and streams decode in Rust. Each fix here teaches the Python side a rule the Rust side already applied.
What changed
- Bytes read like files. On the synchronous bytes route, a JSON number that overflows a float (
1e400) is a decode error instead of an infinity (#1144), and a lone surrogate escape ("\ud800") is a decode error instead of aUnicodeEncodeErrorcrash under either error policy (#1145). JSONL bytes split records at\nonly, so a U+2028 inside a string no longer cuts a record in two, and two records joined by CR, NEL, LS or PS on one line are refused, as the file reader refuses them (#1147). - Your Arrow stream no longer waits on itself.
profile()hung on an Arrow stream whose producer needs the GIL on another thread, such as a PyArrow scanner over a filesystem written in Python. The stream callbacks now run with the GIL released (#1139). - Tiny values keep their spread. Near 1e-200 the squared deviations underflowed, and a column reported a variance of 0.0 with the CV and kurtosis of a constant. The accumulator now holds them at a raised scale; a spread that already fitted reports the same bits as before (#1151).
- Exact no-null rules.
check(max_null_count={"customer_id": 0}),--max-null-count customer_id=0andQualityPolicy::max_null_countdecide on the exact count.max_null_percentage=0decides on the two-decimal value, so one null in 20,001 rows passes it (#1141). - Summaries and errors say more.
python -m dataprof.check --baselinelists the baseline's schema-change diagnostics in its text summary, not only under--json(#1129). A polarsLazyFrameor a PyArrowScannerpassed toprofile()gets an error naming the call that makes it readable,.collect()or.to_reader()(#1136).
The wheels and the locked Rust build carry zstd past RUSTSEC-2026-0391, 0398, 0403 and 0410 (#1131). Rust projects pick the patched crates up with cargo update.
Install
# Python
pip install --upgrade dataprof==0.13.1
# Rust
cargo add dataprof@0.13.1Wheels cover standard CPython 3.11 to 3.15 on Linux, macOS and Windows, as for 0.13.0. Rust 1.96 remains the minimum supported version.
Numbers that move
- JSON bytes with an overflowing number or a lone surrogate escape: under
strictthey raise; underskipa JSONL record is dropped and counted inerror_count, and a JSON document is refused whole. Before, the number profiled as an invalid float and the surrogate crashed the run. - JSONL bytes holding CR, VT, FF, U+001C to U+001E, NEL, LS or PS: record and error counts now match the file reader.
- Numeric columns of very small values, from about 1e-160 down: variance, CV, skewness and kurtosis were zero or drifted, and now match the same values at unit scale. Other columns report the same numbers as 0.13.0.
For a no-null rule past 20,000 rows, move from max_null_percentage=0 to max_null_count=0.
Still known
The 0.13.0 limitations below still hold, except the JSON bytes differences (#1086, #1087) and the spread of underflowing values, fixed here. Decimals whose fraction lies below f64 spacing at a large magnitude still report a variance of zero (#1149).
Full changelog
[0.13.1] - 2026-10-28
Added
- Check: An exact null-count ceiling for no-null rules (#1141) by @AndreaBozzo
Changed
-
Release: Bump to 0.13.1 and generate the changelog by @AndreaBozzo
-
Python: Restore the escaped pairs in the lone-surrogate test (#1148) by @AndreaBozzo
-
Python: Exit the GIL deadlock child without finalizing (#1140) by @AndreaBozzo
Dependencies
-
Bump zstd, zstd-safe and zstd-sys past RUSTSEC-2026-0391/0398/0403/0410 (#1131) by @AndreaBozzo
-
Deps: Bump tokio from 1.53.3 to 1.54.1 (#1112) by @dependabot[bot]
-
Deps: Bump memmap2 from 0.9.11 to 0.9.12 (#1113) by @dependabot[bot]
-
Deps: Bump jsonschema from 0.58.6 to 0.60.0 (#1114) by @dependabot[bot]
-
Deps-dev: Bump ty from 0.0.88 to 0.0.91 (#1115) by @dependabot[bot]
-
Deps: Bump ipython from 8.39.0 to 9.17.1 (#1116) by @dependabot[bot]
-
Deps-dev: Bump ruff from 0.17.1 to 0.17.2 (#1117) by @dependabot[bot]
Documentation
-
Release: 0.13.1 release notes by @AndreaBozzo
-
Correct the null-marker note and the database debug-logging recipe (#1133) by @AndreaBozzo
Fixed
-
Metrics: Keep the spread of values too small to square in f64 (#1151) by @AndreaBozzo
-
Python: Split JSONL bytes at newlines only, as the file reader does (#1147) by @AndreaBozzo
-
Python: Refuse a lone surrogate escape in JSON bytes instead of crashing (#1145) by @AndreaBozzo
-
Python: Refuse a JSON number that overflows a float on the bytes route (#1144) by @AndreaBozzo
-
Python: Release the GIL around Arrow stream producer callbacks (#1139) by @AndreaBozzo
-
Python: Name the lazy source and the call that reads it in refusal errors (#1136) by @AndreaBozzo
-
Check: List baseline diagnostics in the text summary (#1129) by @AndreaBozzo
Release artifacts
Python
- PyPI:
pip install dataprof==0.13.1 - GitHub Releases: Baseline and optimized wheels are attached as release assets
Source
- GitHub automatically attaches source
.tar.gzand.ziparchives for this tag
Rust
- Crates.io:
cargo add dataprof@0.13.1 - The published
dataprofcrate is the stable Rust facade for profiling files, streams, tables, and database queries. - Internal workspace crates are published so the facade can depend on versioned packages.
- This release ships libraries and Python packages; a CLI binary is not part of the release surface.
Versioned documents: full changelog · release notes and migration guide