Skip to content

v2026.08.22 — data-quality check suite

Latest

Choose a tag to compare

@jrosskopf jrosskopf released this 23 Aug 05:49
· 1 commit to main since this release
40ab2d3

Adds a complete data-quality checking workflow to the extension: define checks once, run them all in one call, keep history — all in plain SQL.

New check primitives (each as anofox_tab_* plus a short alias):

  • regex_match — share of values matching a pattern vs thresholds
  • values_in_set — allowed-value membership rate, with sample violations
  • agg_check — avg/min/max/sum/median/stddev of a column vs bounds
  • duplicate_count — duplicates over a column or multi-column key
  • occurrence — highest/lowest frequency of any single value
  • match_rate — cross-table join match rate (referential-integrity check)
  • compliance — share of rows satisfying a custom SQL predicate; expressions are validated at bind time so multi-statement injection payloads fail the binder

Time-aware checks:

  • rel_count_change — daily (distinct) count vs a rolling baseline average
  • metric_anomaly_iqr — daily metric outside Q1/Q3 ± k·IQR of the trailing window; point it at a persisted results table for anomaly detection on your check history
  • rolling_values_in_set — set membership over a trailing date window

Check-suite runner:

  • run_checks(checks_table) reads a SQL table of check definitions and returns one uniform result row per check (run_ts, check_name, check_type, table_name, column_name, identifier, value, lower/upper_threshold, status, message)
  • monitor_only severity (warn instead of fail), per-partition results via identifier_column, ${today}/${yesterday±N} filter tokens, and per-check error isolation (a missing target becomes an error row instead of failing the run)
  • Persist history with plain INSERT INTO dq_results SELECT * FROM run_checks('dq_checks')

Python package anofox-tabular==2026.8.22 (PyPI): wrappers for all new functions plus define_checks/run_checks with optional result-table persistence.

Other changes:

  • DuckDB target bumped to v1.5.5 (Python pin aligned)
  • Suite-runner params are parsed with DuckDB's bundled yyjson — no dependency on the json extension at runtime
  • Windows is excluded on the v1.4.x LTS line (flaky vcpkg MSYS downloads in that toolchain); Windows users are served by the v1.5.x builds
  • Validated at scale: a ~2M-row seeded-defect test in CI, plus the 2.96M-row NYC yellow-taxi Jan-2024 dataset where the checks surface the real defects (124 duplicate trips, negative fares, 4.7% NULL passenger_count)