Skip to content

dataprof v0.13.1

Choose a tag to compare

@github-actions github-actions released this 28 Oct 16:55
· 130 commits to master since this release

0.13.0 shipped with a confession in its known limitations: some JSON bytes behaved differently from the same bytes in a file. Three days later that list is shorter. This is a patch release: nothing removed, one rule added, and the wheels rebuilt on patched zstd.

The 0.13 audits profiled the same bytes through every route and compared the answers. The routes that still disagreed were on the synchronous bytes path, which decodes JSON with Python's json module, while files, async reads and streams decode in Rust. Each fix here teaches the Python side a rule the Rust side already applied.

What changed

  • Bytes read like files. On the synchronous bytes route, a JSON number that overflows a float (1e400) is a decode error instead of an infinity (#1144), and a lone surrogate escape ("\ud800") is a decode error instead of a UnicodeEncodeError crash under either error policy (#1145). JSONL bytes split records at \n only, so a U+2028 inside a string no longer cuts a record in two, and two records joined by CR, NEL, LS or PS on one line are refused, as the file reader refuses them (#1147).
  • Your Arrow stream no longer waits on itself. profile() hung on an Arrow stream whose producer needs the GIL on another thread, such as a PyArrow scanner over a filesystem written in Python. The stream callbacks now run with the GIL released (#1139).
  • Tiny values keep their spread. Near 1e-200 the squared deviations underflowed, and a column reported a variance of 0.0 with the CV and kurtosis of a constant. The accumulator now holds them at a raised scale; a spread that already fitted reports the same bits as before (#1151).
  • Exact no-null rules. check(max_null_count={"customer_id": 0}), --max-null-count customer_id=0 and QualityPolicy::max_null_count decide on the exact count. max_null_percentage=0 decides on the two-decimal value, so one null in 20,001 rows passes it (#1141).
  • Summaries and errors say more. python -m dataprof.check --baseline lists the baseline's schema-change diagnostics in its text summary, not only under --json (#1129). A polars LazyFrame or a PyArrow Scanner passed to profile() gets an error naming the call that makes it readable, .collect() or .to_reader() (#1136).

The wheels and the locked Rust build carry zstd past RUSTSEC-2026-0391, 0398, 0403 and 0410 (#1131). Rust projects pick the patched crates up with cargo update.

Install

# Python
pip install --upgrade dataprof==0.13.1

# Rust
cargo add dataprof@0.13.1

Wheels cover standard CPython 3.11 to 3.15 on Linux, macOS and Windows, as for 0.13.0. Rust 1.96 remains the minimum supported version.

Numbers that move

  • JSON bytes with an overflowing number or a lone surrogate escape: under strict they raise; under skip a JSONL record is dropped and counted in error_count, and a JSON document is refused whole. Before, the number profiled as an invalid float and the surrogate crashed the run.
  • JSONL bytes holding CR, VT, FF, U+001C to U+001E, NEL, LS or PS: record and error counts now match the file reader.
  • Numeric columns of very small values, from about 1e-160 down: variance, CV, skewness and kurtosis were zero or drifted, and now match the same values at unit scale. Other columns report the same numbers as 0.13.0.

For a no-null rule past 20,000 rows, move from max_null_percentage=0 to max_null_count=0.

Still known

The 0.13.0 limitations below still hold, except the JSON bytes differences (#1086, #1087) and the spread of underflowing values, fixed here. Decimals whose fraction lies below f64 spacing at a large magnitude still report a variance of zero (#1149).

Full changelog

[0.13.1] - 2026-10-28

Added

Changed

  • Release: Bump to 0.13.1 and generate the changelog by @AndreaBozzo

  • Python: Restore the escaped pairs in the lone-surrogate test (#1148) by @AndreaBozzo

  • Python: Exit the GIL deadlock child without finalizing (#1140) by @AndreaBozzo

Dependencies

Documentation

Fixed

  • Metrics: Keep the spread of values too small to square in f64 (#1151) by @AndreaBozzo

  • Python: Split JSONL bytes at newlines only, as the file reader does (#1147) by @AndreaBozzo

  • Python: Refuse a lone surrogate escape in JSON bytes instead of crashing (#1145) by @AndreaBozzo

  • Python: Refuse a JSON number that overflows a float on the bytes route (#1144) by @AndreaBozzo

  • Python: Release the GIL around Arrow stream producer callbacks (#1139) by @AndreaBozzo

  • Python: Name the lazy source and the call that reads it in refusal errors (#1136) by @AndreaBozzo

  • Check: List baseline diagnostics in the text summary (#1129) by @AndreaBozzo

Release artifacts

Python

  • PyPI: pip install dataprof==0.13.1
  • GitHub Releases: Baseline and optimized wheels are attached as release assets

Source

  • GitHub automatically attaches source .tar.gz and .zip archives for this tag

Rust

  • Crates.io: cargo add dataprof@0.13.1
  • The published dataprof crate is the stable Rust facade for profiling files, streams, tables, and database queries.
  • Internal workspace crates are published so the facade can depend on versioned packages.
  • This release ships libraries and Python packages; a CLI binary is not part of the release surface.

Versioned documents: full changelog · release notes and migration guide