Skip to content

dataprof v0.10.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 24 Jul 22:26
· 37 commits to master since this release
v0.10.0
b7b139f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

0.10.0 is a correctness-and-control release. Its theme is simple: a profiling
option must take effect, damaged input must leave evidence, and a score must
describe only what dataprof actually assessed. That sounds obvious; the
pre-release dogfooding pass found several places where different engines,
transports, or bindings had drifted away from that contract.

This release makes sampling real across every supported path, forwards resource
controls to the engines that use them, closes major parser gaps across
file/bytes/async inputs, adds Validity and Precision to the quality model, and
replaces vacuous-perfect dimension scores with assessed-only aggregation.
Reports also gain stronger execution provenance, exact semantic-hint binding
evidence, discoverable build capabilities, and clearer errors.

The result is intentionally more honest, even where honesty changes a number or
turns a formerly accepted call into an error.

Install

# Python
pip install --upgrade dataprof==0.10.0

# Rust
cargo add dataprof@0.10.0

Python 3.10+ and Rust 1.96+ remain the supported minimums. dataprof 0.10.0 ships
libraries and Python packages; there is no CLI binary.

Upgrade checklist

If you rely on… What changed What to do
quality_score thresholds The aggregate now scores only assessed dimensions, includes every sub-metric, and has seven default dimension weights. An unassessable report returns None in Python/Rust report helpers instead of a fabricated 100. Re-run representative profiles and re-baseline every quality gate. Inspect assessed_dimensions() and the underlying facts beside the aggregate.
SamplingStrategy.importance(...) The signature is now importance(weight_column, weight_threshold). Sampling is actually applied on supported CSV/async paths; unsupported readers reject it instead of profiling everything. Name the numeric weight column explicitly and handle ValueError for unsupported source/engine combinations.
chunk_size or memory_limit_mb chunk_size consistently means bytes, and both controls now reach the engines that document them. Remove any row-based interpretation of chunk_size; choose it as an I/O/memory granularity.
broad except RuntimeError blocks Missing files, invalid input/configuration, permissions, and I/O now map to idiomatic Python exceptions. Malformed CSV and JSON both raise ValueError in strict paths. Catch FileNotFoundError, ValueError, PermissionError, or OSError as appropriate.
semantic hints Unknown or provably inert positive_columns / temporal_columns hints now raise instead of disappearing silently. Correct stale column names and request the Quality metric pack when using value-driven hints.
direct Rust metrics methods MetricsCalculator::calculate_{comprehensive,bifurcated}_metrics_with_positive_columns gained row_duplicates: Option<RowDuplicateSummary>. Pass None to keep the previous behavior, or provide the engine's row-duplicate summary.

Release highlights

  • Sampling is deterministic and stateful. Fixed-size strategies hold the
    final sample; progressive sampling measures the data; multi-stage composition
    is validated before a scan begins.
  • Execution metadata is internally consistent. Hard row caps are exact,
    fully consumed sources are exhausted, byte counts include syntax, and ragged
    CSV recovery is visible.
  • The hardened parser paths agree across transports. Truncated JSONL
    records, malformed JSON arrays, async delimiters, and malformed CSV now follow
    explicit strict/tolerant policies. Remaining JSONL boundary choices are listed
    under known limitations.
  • Quality scores explain their denominator. Validity and Precision bring the
    model to seven dimensions, configurable weights renormalize over assessed
    dimensions, and full-stream duplicate tracking feeds uniqueness.
  • Errors and capabilities are usable API surfaces. Callers can discover what
    the installed build supports and handle failures by category without parsing
    generic runtime strings.

The sections below contain the migration details and the reasoning behind each
compatibility-sensitive change.

Security and dependency integrity

Security auditing covers every published feature graph

cargo deny ran against the default build only. Optional features are still
published features — the database connectors pull in the whole SQLx client
stack, and the async and Parquet features pull in a TLS stack — so CI reported
green while a shipped feature graph carried an advisory nobody had reviewed. The
security workflow now audits both the default and the complete feature graph,
and the two disagreed on more than advisories: the licence check had also never
seen ISC or CDLA-Permissive-2.0, both reached only through the TLS stack and
both permissive. They are now explicitly allowed.

Dependency updates that came with it:

  • arrow and parquet 58 → 59, which removes the thrift dependency entirely
    and with it GHSA-2f9f-gq7v-9h6m
    (excessive allocation size). No API changes were required.
  • rand 0.8.5 → 0.8.7 (transitive, via SQLx), clearing
    GHSA-cq8v-f236-94qc.
  • spin 0.9.8 → 0.9.9, replacing the yanked 0.9.8 release.
  • Pygments 2.19.2 → 2.20.0 in the Python environment, clearing
    GHSA-5239-wwwm-4pmq.
    Dependabot was not watching the Python dependencies at all, so nothing would
    have proposed this; it now covers uv.lock alongside Cargo and Actions.

RUSTSEC-2023-0071 (Marvin attack in rsa) is recorded as not applicable,
with the reasoning and a review date in
docs/SECURITY.md. In short: dataprof reaches
rsa only through sqlx-mysql, whose sole use of it encrypts a password with
the server's public key. It never performs a private-key operation and holds
no private key, so there is nothing for a private-key timing attack to target —
and the code path is skipped entirely over TLS, which is how dataprof builds
SQLx. No fixed release of rsa exists. The disposition is recorded in both
deny.toml and .cargo/audit.toml, so cargo deny and cargo audit agree.

Execution controls that hold

Sampling strategies actually sample

sampling= was undependable across engines and inputs. The documented default
call, dp.profile(path, sampling=...), dropped the strategy and profiled
everything; forcing the incremental or async engine returned all rows, no rows,
or an arbitrary subset depending on which strategy was chosen. The cause was
that both engines asked a stateless helper about each row, which built fresh
state every time and never passed the row's values — so stateful strategies
behaved as if every row were the first, and row-aware ones could never match.

Every strategy is now applied by one shared sampler that holds its state for the
whole scan and can read the row it is deciding on.

Fixed-size samples are drawn at the end of the source. Whether a row belongs
in a uniform sample of n is not known until the stream ends — a row selected
early can be evicted later — and streaming statistics cannot be retracted. So
random(n) and reservoir(n) now hold their candidate rows and compute the
profile from the surviving sample, at a cost of n rows of memory. They give
the same guarantee (a uniform sample of exactly n rows, or the whole source if
it is shorter); both names are kept because both are familiar. Every other
strategy decides per row and adds no memory.

Two strategies were redefined, because their names promised more than their
arithmetic delivered.

  • progressive(initial, confidence_level, max) measured "confidence" as
    1 - 1/sqrt(n), which ignores the data entirely and cannot reach 0.95 below
    400 rows — so for any smaller max_size the parameter did nothing and the
    strategy just took max_size rows. It now measures the relative standard
    error     of each numeric column's mean and stops once every one is within
    1 - confidence_level of its mean. Low-variance data stops early; volatile
    data runs to max_size. A source with no numeric columns has no measurable
    precision and samples max_size rows.
  • importance(weight_threshold) scored rows with a built-in heuristic under
    which any complete row passed a threshold of 0.5, and applied no
    inverse-probability correction. It now takes the column to weigh on:
    importance(weight_column, weight_threshold) keeps rows whose value in that
    column is a number at or above the threshold. This is a breaking signature
    change.     It is a filter, so the resulting profile describes the rows that met
    the threshold, not the source as a whole.

Multi-stage composition is now defined. Streaming stages act as filters in
sequence, and at most one fixed-size stage may appear, last — it draws its
sample from whatever the filters passed. Two fixed-size stages, or a filter
after one, are refused before reading rather than silently dropping stages.

No path ignores the option any more. engine="auto" routes a sampled CSV
run to the engine that can honour it. The columnar engine, the JSON and Parquet
readers, and synchronous bytes input raise instead of returning a full profile
under a sampling request.

sampling_applied and sampling_ratio now describe the rows that reached the
profile, and a strategy that happened to keep every row no longer reports itself
as sampling. Sampling also no longer marks a fully read source as unexhausted —
that field answers whether the source ran out, which is what truncation_reason
accompanies.

Note that sampling bounds the cost of analysis, not of reading: a uniform
sample requires seeing the whole source. Pair it with a StopCondition to bound
I/O.

Execution controls take effect, and execution metadata tells the truth

chunk_size and memory_limit_mb were accepted and then dropped on the paths
that document them, so a resource control could be silently ineffective. Both
now reach the engine that does the work:

  • The incremental engine honors an explicitly configured chunk_size instead of
    always deriving one from the memory limit and file size.
  • memory_limit_mb is forwarded on the incremental and default (auto) paths,
    not only on the explicit columnar and async ones.

chunk_size is measured in bytes, on every engine and in every binding. The
type documented rows while the async reader treated the value as bytes and the
incremental engine ignored it; bytes is now the single unit, because it is what
actually bounds the working set. Chunk size never changes the result of a
complete scan — only read granularity, progress cadence, and the points at which
chunk-level stop conditions are evaluated. Async callers who passed
ChunkSize::Fixed see effectively no change, since that path already divided
the value by an assumed row width.

Stop conditions and the metadata they produce were also inconsistent:

  • Row caps are hard caps. dataprof.asyncio evaluated max_rows per chunk,
    so a request for 123 rows could return 200 while the report named the limit of
    123. Async CSV, JSON and JSONL now stop exactly at the cap.
  • A condition met on the final chunk is not a truncation. A confidence
    threshold or quality_sample() preset satisfied by the last chunk of a fully
    read file reported source_exhausted: false with a truncation reason, making
    a complete profile indistinguishable from a bounded one. The same holds for a
    row cap equal to the row count.
  • Schema stability keeps its counters. Stopping on schema_stable reset the
    stop evaluator to suppress a duplicate reason, discarding the accumulated byte
    count with it and reporting bytes_consumed: 0 on a scan that had read
    thousands of bytes.
  • Async byte counts are no longer short. They were summed from parsed
    fields, which exclude delimiters, quotes and line endings, so even a complete
    scan reported fewer bytes than the source held. CSV now counts from the
    parser's own byte position, and a scan that reaches the end of a source of
    known length reports that source's full size.

Byte caps remain chunk-boundary caps: bytes_consumed may exceed MaxBytes by
at most one chunk, a bound the caller controls through chunk_size. Row caps
have no such allowance.

Python's dataprof.asyncio.profile_bytes() now preserves the buffer's known
length in the async source metadata. Complete JSON and JSONL byte scans therefore
report the exact input length in bytes_consumed, rather than an estimate
derived from parsed values; bounded scans still report only the bytes they
actually read.

rows_processed, bytes_consumed, source_exhausted and truncation_reason
are now covered by invariant tests on both the sync and async paths — a
truncated scan is exactly a non-exhausted one, and an exhausted scan accounts
for every byte of its source.

Parser behavior aligned across hardened paths

Ragged CSV rows leave a signal instead of vanishing

A CSV row whose field count differs from the header — extra trailing fields, or
missing ones — is still recovered (extra fields dropped, missing fields padded
to null) so profiling continues, but that recovery is no longer silent.
execution.ragged_row_count reports how many rows were ragged; it is 0 for a
cleanly parsed file. Previously such files reported error_count: 0 and a
perfect consistency score, answering "did parsing silently go wrong?" with a
confident and wrong "no".

The count is exposed on the Python ProfileReport as report.ragged_row_count
and in to_dict()["execution"]["ragged_row_count"]. It is an additive report
field: reports written before this release deserialize with ragged_row_count
of 0.

The async reader follows the same policy
(#462): byte streams and
URLs recover ragged rows and report the same count, so the transport can no
longer launder a broken source into a clean-looking report. csv_flexible=False
now also reaches that path — previously it was accepted and ignored there — and
rejects the first ragged record instead of repairing it.

Scope: the count is surfaced by the incremental engine, which drives the default
CSV path, and by the async reader. Byte inputs to the synchronous profile()
still reject rather than recover, since they are read without the flexible
engine; write the data to a file to profile it leniently.

The explicitly selected columnar (Arrow) engine is not yet covered. It rejects a
row with extra fields, but pads a row with missing fields to null and still
reports ragged_row_count: 0 — so engine="columnar" remains a silent-clean
path for short rows. Prefer the default engine="auto" when a source may be
structurally broken; the gap is tracked in
#470.

Ragged rows do not yet influence the consistency dimension's score.

Truncated JSONL records are never a clean EOF

An incomplete final JSONL object was silently discarded on file and async
inputs because serde_json classifies both a clean end of input and a partial
value as EOF. That made tolerant mode report error_count: 0, and even
jsonl_on_error="strict" returned a successful profile. The scanners now
distinguish an empty or whitespace-only remainder from a value that started but
did not finish. Tolerant mode skips and counts the partial record; strict mode
raises ValueError, matching synchronous bytes input.

Async CSV detects its delimiter instead of assuming a comma

csv_delimiter was accepted and ignored on every async path, which always
parsed on commas. A semicolon- or tab-separated stream therefore collapsed into
a single column and profiled as perfectly clean. The async reader now honors an
explicit csv_delimiter, and when none is given it detects one from the head of
the stream using the same sample size and scoring as the file path — so the same
bytes yield the same columns whether they are read from disk, from memory, or
off a URL.

Streaming JSON arrays enforce their container grammar

The file and async JSON readers streamed each array value correctly but treated
commas as optional whitespace and stopped at either ] or EOF. Missing,
leading, doubled, and trailing commas; a missing closing bracket; trailing
garbage; and a second top-level value could all produce a clean profile.
Container parsing now follows the JSON array state machine and validates the
closing bracket and trailing input. Tolerant mode retains a valid prefix but
increments error_count; strict mode raises ValueError. A scan intentionally
bounded by max_rows still does not validate values it did not read.

Malformed CSV raises ValueError, like malformed JSON

A CSV parse failure reached Python as a RuntimeError while the equivalent JSON
failure raised ValueError, so callers could not catch bad input as one
category. Malformed data of either format is now a ValueError.

Relatedly, a rejection under csv_flexible=False keeps its own diagnostic. The
auto engine used to retry the file under its fallback parser and report
All engines failed: ... with both parsers' messages; asking for strict parsing
means opting out of recovery, so the original error — naming the row and the
expected field count — is returned directly.

Reports that explain what was assessed

Locale patterns require locale evidence before becoming report claims

Ambiguous locale-specific shapes remain available in each column's detailed
patterns evidence, but no longer appear as a top pattern in text, Markdown,
HTML, tabular, or LLM-oriented summaries unless their confidence is at least
0.5. For example, a five-digit order ID column without a configured locale can
still show both Italian CAP and US ZIP as low-confidence candidates, but neither
is presented as the column's semantic type.

An explicit locale= is now case-insensitive and strict: its matching patterns
receive enough evidence to be reportable at a strong match rate, while patterns
for other locales are suppressed even when the broad regex matches every row.
Coordinate validation also distinguishes compact latitude/longitude pairs from
decimal-comma numbers such as 1.234,56. More generally, a pattern is no longer
returned when its semantic validator rejects every regex match, and such a
candidate cannot suppress another pattern during overlap resolution. This
resolves #429.

Errors preserve source context and map to idiomatic Python exceptions

Failure diagnostics are now consistent about what failed, where, and what to do
next. This is compatibility-sensitive: error messages and, on the Python
side, exception types have changed.

  • Real paths, never 'unknown'. The context-free From<io::Error>,
    From<anyhow::Error>, and From<csv::Error> conversions no longer fabricate
    a FileNotFound { path: "unknown" }; they keep the original message and let
    call sites that hold the path attach it. Profiler::analyze_file now fails
    fast with a FileNotFound naming the real path when the file is missing.
  • Honest supported-format claims. UnsupportedFormat lists only the formats
    the running build can actually read — Parquet appears only when the parquet
    feature is enabled — instead of the fixed CSV, JSON, JSONL string that
    omitted Parquet.
  • Structured suggestions. DataProfilerError::suggestion() exposes the
    actionable next step as data, so callers no longer have to parse it out of the
    formatted message.
  • Credential redaction. Database connection and query errors scrub
    scheme://user:password@host userinfo before the message is stored, and the
    "invalid connection string" errors report the detected scheme instead of
    echoing the raw string.
  • Idiomatic Python exceptions. File-based entry points now raise
    FileNotFoundError (missing file), ValueError (unsupported format, invalid
    config, unbindable semantic hints), and PermissionError / OSError (I/O),
    instead of wrapping every failure in a generic RuntimeError. except RuntimeError: blocks that relied on the old behavior need updating.

Installed capabilities are discoverable

dataprof.capabilities() returns an immutable snapshot of the current build:
local formats, compiled pandas/polars/Arrow interoperability, installed optional
Python packages, async and URL support, remote Parquet support, database
availability, compiled connectors, and the package version. Discovery imports
no heavyweight optional dependency and performs no file, database, or network
operation.

Validity and Precision join the quality model

Two new selectively requestable dimensions are available in Rust and Python:

  • Validity measures conformance to a confidently detected semantic pattern.
    It stays unassessed when pattern detection did not run or no pattern has
    enough confidence, rather than assuming values are valid.
  • Precision measures consistency of effective decimal places within each
    floating-point column. It reports deviation from the observed modal scale;
    it does not infer a business-required number of decimal places.

Both dimensions participate in assessed_dimensions(), per-dimension scores,
streaming provenance, report serialization, and Python nested quality dicts.
Adding them changes the default score weights to 0.25 completeness, 0.20
consistency, 0.15 uniqueness, 0.15 accuracy, 0.10 timeliness, 0.10 validity,
and 0.05 precision. Re-baseline aggregate-score gates; underlying facts remain
individually inspectable.

Quality score weights are configurable

The overall score's relative dimension weights now live in
IsoQualityConfig::score_weights. MetricsCalculator::with_thresholds(...)
copies them into each QualityMetrics result, so custom scores remain
reproducible after report serialization. Weights renormalize over the
dimensions that were actually assessed.

The documentation now describes ISO 8000 and ISO/IEC 25012 as sources for the
quality-dimension concepts. Dataprof's aggregate score and its weights are a
configurable project formula, not an ISO-mandated or certified score.

The quality score now only scores what was actually assessed

quality_score changes value for almost every dataset. The facts
(per-dimension counts and ratios) are unchanged; how they aggregate is not.

Previously, a dimension with nothing to assess counted as perfect: no numeric
columns meant accuracy 100, no date columns meant timeliness 100, no id-named
column meant uniqueness 100. A text-only CSV could not score below 70 no matter
how bad it was. And the violation counts (duplicate rows, format violations,
encoding issues, range violations, future dates, temporal violations) never
influenced the score at all.

Now:

  • Each dimension records how much data it examined (total_cells,
    values_checked, rows_checked, numeric_values_checked,
    date_values_checked). A dimension that examined nothing is excluded
    and the weights renormalize over the assessed dimensions
    (QualityMetrics::assessed_dimensions(), Python
    quality.assessed_dimensions() / quality.dimension_scores()).
  • Every sub-metric now feeds its dimension score: duplicate rows drive
    uniqueness, format violations and encoding issues drive consistency, range
    violations drive accuracy, future dates and temporal violations drive
    timeliness. Completeness is the mean of cell-level and row-level
    completeness.
  • Timeliness scoring assesses confidently inferred date columns by default.
    Explicit temporal_columns hints add columns that inference cannot identify,
    such as mixed-format strings. Non-null values that fail strict calendar
    parsing are reported as invalid_date_values and reduce the timeliness score
    instead of silently disappearing from its denominator.
  • UniquenessMetrics.key_column names the column key_uniqueness describes;
    when no key column is identified, key uniqueness carries no signal instead
    of "assume perfect".
  • Explicit identifier_columns hints now select the column used by
    key_uniqueness. Without a hint, key-name inference matches complete words
    such as id, key, and pk (including snake/kebab/camel case) instead of
    substring false positives such as paid, valid, or monkey.
  • The duplicate-row scan refuses per-column samples it cannot prove
    row-aligned (null-stripped or reservoir-evicted columns) instead of
    comparing unrelated values.
  • report.quality_score() returns None when nothing was assessable (empty
    dataset, or a report serialized by an older version) instead of a
    fabricated 100. overall_score() on raw metrics returns 0.0 in that case —
    check assessed_dimensions() to tell the difference.

What to do: re-baseline any thresholds built on quality_score. Scores
move down or stay; datasets whose old score leaned on vacuous-perfect
dimensions drop the most. At that stage the five-dimension weights remained
0.30 / 0.25 / 0.20 / 0.15 / 0.10; the Validity and Precision change above
subsequently expands and rebalances them. The formula remains dataprof's own,
not an ISO-mandated one.

Duplicate rows are now counted over the full stream

The CSV, JSON, and streaming engines track row identity while they read:
every record's fields fold into a length-prefixed signature fed to the same
exact-then-HLL distinct estimator that backs unique_count. As a result:

  • duplicate_rows covers every row of the source — including rows with
    null values, which the old sample-based scan could never see — with
    rows_checked reporting the full row count.
  • Below ~10,000 distinct rows the count is exact. Beyond that the distinct
    estimator spills to its HLL sketch and the derived duplicate count is an
    estimate (~1% relative error on distincts), flagged by the new
    UniquenessMetrics.duplicate_rows_approximate field and filed as
    non-exact provenance.
  • Engines without a row tracker (Parquet record batches, DataFrames, Arrow)
    keep the alignment-guarded sample scan; when the sample cannot be proven
    row-aligned, duplicate_rows stays not-assessed rather than wrong.

MetricsCalculator::calculate_{comprehensive,bifurcated}_metrics_with_positive_columns
gained a row_duplicates: Option<RowDuplicateSummary> parameter; pass None
to keep the previous behavior.

Semantic hints are validated, not silently dropped

A semantic hint (positive_columns, identifier_columns, temporal_columns)
is the user's chosen alternative to overconfident inference, so a hint that
cannot bind is now an error instead of a silent no-op:

  • A hint that names a column not in the schema raises, listing the unmatched
    names and the available columns. In Python this is a ValueError.
  • A hint that names a real column but binds to no value over the full data —
    a positive hint on a column with no numeric values, a temporal hint on a
    column with no dates — also raises. Identifier hints coerce the column's type,
    so they bind to any existing column and are only rejected for an unknown name.
  • Mixed columns still bind: a temporal hint on a column where only some values
    parse as dates is assessed, not rejected.

Reports gained semantic_hint_bindings, per-column evidence of how each hint
bound (column, kind, checked_values, matched_values, exact).
Value-driven hints are counted by bounded-memory accumulators over the full
processed stream, even when quality metrics use a reservoir sample. Their
evidence is therefore exact, and an inert hint is rejected consistently on
large streamed sources without risking a false positive when a match exists
outside the retained sample. Supplying positive_columns or
temporal_columns without the Quality metric pack is also an error because
those hints would have no consumer; identifier_columns remains usable because
it affects column typing. The field is additive; older readers ignore it.

Known limitations carried into 0.11.0

The dogfooding pass also found issues that need an explicit product decision or
larger implementation change. They are documented and scheduled rather than
hidden in 0.10:

  • The columnar CSV engine can pad a short row without incrementing
    ragged_row_count; the default incremental engine reports it correctly
    (#470).
  • JSON and JSONL still need one cross-transport contract for zero-field records,
    source field order, and physical record boundaries
    (#463,
    #465,
    #486).
  • Python Parquet byte buffers still require the pandas extra even though local
    Parquet files work in the dependency-free wheel
    (#461).
  • A sparse optional column can make strict all-column
    complete_records_ratio unhelpful; inspect cell completeness and the named
    null-heavy columns alongside it
    (#436).

Full changelog

[0.10.0] - 2026-07-24

Added

Changed

Dependencies

Documentation

Fixed

Release artifacts

Python

  • PyPI: pip install dataprof==0.10.0
  • GitHub Releases: Baseline and optimized wheels are attached as release assets

Source

  • GitHub automatically attaches source .tar.gz and .zip archives for this tag

Rust

  • Crates.io: cargo add dataprof@0.10.0
  • The published dataprof crate is the stable Rust facade for profiling files, streams, tables, and database queries.
  • Internal workspace crates are published so the facade can depend on versioned packages.
  • This release ships libraries and Python packages; a CLI binary is not part of the release surface.

Versioned documents: full changelog · release notes and migration guide

Contributors

AndreaBozzo, dependabot, and floze-the-genius
Assets 58
Loading