v0.16.0

What's Changed

• Added a pluggable actions and alerting subsystem (#1289). DQX now supports extensible actions that run when checked data violates an optional condition evaluated against the summary metrics produced by DQMetricsObserver. The built-in DQAlert action can send notifications to Slack, Microsoft Teams, a generic HTTPS webhook, or the log, so pipelines can react to data quality regressions without custom plumbing. You can create your own custom actions as well, and custom alerting is possible via the callback destination, which invokes an in-process Python callable for each alert.
• Added an MCP (Model Context Protocol) server for DQX (#1252). The server exposes DQX's data quality capabilities as tools that any MCP-compatible AI agent (Claude, Genie Code, Cursor, Mosaic AI) can discover and orchestrate. It runs as a Databricks App with on-behalf-of (OBO) authentication, so all data access is governed by the calling user's Unity Catalog permissions.
• Added support for summary metrics in Lakeflow Declarative Pipelines (LDP/DLT) (#1301). A new DQEngine.compute_summary_metrics(...) produces the same row counts, per-check breakdown, and custom observer metrics as a lazy aggregation over the results DataFrame, so metrics can be computed inside Spark Declarative Pipelines where the observer- and streaming-listener-based paths cannot be used.
• Added the aggr_matches_dataset dataset-level check (#1309). The check compares an aggregate metric (row count by default, or any curated/built-in aggregate) computed on the checked DataFrame against the same aggregate computed on a reference (upstream) table, enabling reconciliation-style validations against a source of truth.
• Added the has_no_gaps_per_time_window dataset-level check (#1370). It detects gaps in a time series — windows of a configurable size that contain no rows between windows that do — with optional grouping and trailing-gap handling.
• Added semantic validation of rulesets (#1203). A new ChecksSemanticValidator inspects declarative check metadata and reports duplicate rules (same function, arguments, criticality, and filter) and conflicting rules, surfacing authoring mistakes before checks run.
• Added the has_valid_string_case row-level check (#1347). It validates consistent string casing with upper, lower, title, and sentence modes, casting non-string columns to strings before comparison.
• Added the is_valid_national_id row-level check (#1346). It validates national identification numbers per country (default US), covering format, ranges, and obvious structural errors; it does not verify that a number was actually issued.
• Added the is_valid_currency_code row-level check (#1368). It validates values against ISO 4217 currency codes, supporting both the three-letter alphabetic (e.g. USD) and three-digit numeric (e.g. 840) representations via code_format.
• Added the is_valid_country_code row-level check (#1369). It validates values against ISO 3166-1 country codes in alpha-2 (default), alpha-3, or numeric form via code_format.
• Added the is_valid_language_code row-level check (#1403). It validates values against ISO 639 language codes in alpha-2 (ISO 639-1) or alpha-3 (ISO 639-3) form.
• Added the is_valid_subdivision_code row-level check (#1404). It validates values against ISO 3166-2 country subdivision codes (e.g. US-CA, GB-ENG), with optional cross-column country consistency via country_column.
• Added the is_valid_uuid row-level check (#1436). It validates values against the canonical RFC 9562 UUID string form (case-insensitive), mirroring the other pure pattern-match checks.
• Added a profiler profile builder for the has_no_outliers check (#1317). The profiler can now generate a has_no_outliers check, and the MAD-based calculations and profiler defaults were refactored into shared constants. Disabled by default to retain existing performance.
• Added Databricks runtime requirements verification (#1278). DQX now verifies the minimum Databricks environment version before executing checks (including geospatial checks with specific runtime requirements), while preserving backward compatibility.
• Migrated configs and rules to Pydantic v2 (#1259, #1302). DQX data models and the declarative checks path moved from dataclasses to Pydantic v2 for automatic validation and simpler YAML/JSON (de)serialization. Behavior is preserved: callers never see raw pydantic ValidationError — every entry point still raises DQX error types with the pre-migration message format.
• Persisted full check metadata across table storage backends (#1319). Checks saved to Delta or Lakebase now faithfully round-trip all logical fields (including message_expr and typed user_metadata), matching the File/Volume backends. See Breaking Changes for the user_metadata at-rest encoding change on the Delta backend.
• Guarded LLM-generated sql_query rules against unsafe SQL (#1275). Both LLM-assisted rule-generation paths now drop any generated sql_query rule whose query contains unsafe (DML/DDL) SQL before returning it to the caller.
• Guarded check filter and row_filter against unsafe SQL (#1303). All filter compile sites now route through a shared safe_filter_expr helper that rejects destructive SQL keywords, and a check with an unsafe filter is treated as skipped rather than failing the run.
• Decoupled AI-assisted rule generation from Spark (#1422). Rule generation no longer requires an active Spark session — column metadata is fetched via the Databricks SDK, and Spark sessions are created lazily only when required.
• Added a public parameter-order contract for built-in checks (#1408). A test now pins the intended public parameter order of every registered check in check_funcs.py, making any reordering that would break positional callers visible during review.
• DQX Studio: overhauled rule authoring with data-contract and AI generation, insights, and internationalization (#1262). The app can import ODCS v3.x data contracts and generate predefined, property-based, and natural-language rules via a new "from contract" UI, alongside richer authoring and insights.
• DQX Studio: added a French (fr) locale (#1330). French joins the existing English, Brazilian Portuguese, Italian, and Spanish translations.
• DQX Studio: simplified deployment to a natively declarative bundle (#1300). Legacy post-deploy grant scripts and one-time resource binding were removed in favor of a natively declarative Databricks Asset Bundle deploy.
• Deployed the MCP server with native DAB grants (#1430). The MCP server's Unity Catalog grants are now native bundle declarations and the post-deploy setup job was dropped, so make mcp-deploy is a single end-to-end command.
• Made telemetry best-effort so it can never fail a job (#1401). log_telemetry no longer makes a blocking per-check control-plane call that could raise a TimeoutError and terminate Structured Streaming jobs; telemetry is now non-throwing, deduplicated per process, and bounded by a short timeout.
• Stopped reconfiguring the root logger on import (#1359). Importing DQX no longer calls blueprint's install_logger(), which previously removed existing root handlers and overwrote the logging configuration of applications using DQX as a library.
• Fixed filtered is_unique violations (#1442). is_unique now requires the current row to match its filter before reporting a duplicate, so unfiltered rows sharing a key with filtered rows are no longer falsely flagged.
• Fixed aggregate checks over * combined with a row filter (#1453). Dataset-level aggregate checks that aggregate over * with a row_filter no longer raise INVALID_USAGE_OF_STAR_OR_REGEX when constructed with F.col("*"); unfiltered count/count_distinct over * continue to work, and unsupported star/aggregate combinations now raise a clear InvalidParameterError.
• Fixed row-level pattern checks accepting a trailing newline (#1454). is_valid_email, is_valid_ipv4_address, is_valid_uuid, is_valid_national_id, and the is_ipv4_address_in_cidr value path now reject values with a trailing newline, which Java/Spark rlike previously accepted because $ also matches before a final line terminator.
• Fixed null-safe result joins (#1356). Custom SQL and grouped-check results are now reattached with null-safe key matching, preserving violations for rows and groups with null key components.
• Fixed NULL group matching in time-window gap checks (#1387). has_no_gaps_per_time_window now preserves gap violations for groups with null key components.
• Fixed non-deterministic fingerprints for set-valued arguments (#1345). Set and frozenset rule arguments are canonicalized after normalization, so rule fingerprints are stable regardless of set iteration order.
• Fixed validation of columns requiring SQL identifier escaping (#1342). Bare column names that need escaping (e.g. Customer Name) now validate via a two-pass fallback.
• Preserved fractional seconds in timestamp profiles (#1399). The profiler now preserves microsecond precision when converting timestamp min/max aggregates to profile values for TimestampType and TimestampNTZType.
• DQX Studio: fixed the rule source value for rules imported through the app (#1318).
• DQX Studio: fixed installation on Windows (#1327). build_app.py now appends the .cmd suffix to Node binaries on Windows, and CI sweeps orphaned jobs.
• Reported MCP server integration test coverage (#1429). The MCP integration suite now runs on every pull request and installs DQX built from the repository (production keeps the published pin), so a change anywhere in the library is exercised end to end.
• Fixed the set_utc_timezone test fixture to actually apply UTC (#1402).
• Added documentation versioning tags (#1388). User-guide pages and subsections can now be tagged with a feature lifecycle stage and the version a feature was introduced.
• Added a Genie Space connection guide to the Quality Dashboard documentation (#1263).
• Simplified tolerance docstring examples in the check functions (#1254).
• Fixed data contract examples in the documentation (#1250).

BREAKING CHANGES!

• is_in_list, is_not_in_list, and is_not_null_and_is_in_list now resolve their allowed / forbidden string values as column expressions (consistent with the comparison checks), not string literals. A bare string is interpreted as a column reference, a numeric string (e.g. "3") is parsed as a number, and an ISO-date string (e.g. "2024-01-01") as a date. To match a string literal, single-quote the value (e.g. 'value') or wrap it in F.lit("value"). Existing checks that relied on bare strings being treated as literals must quote them. (#1419)
• user_metadata saved through the Delta table storage backend is now JSON-encoded at rest to preserve non-string types through the MAP<STRING, STRING> column. Save→load via DQX is transparent (you get the original typed value back), but the stored representation changes: direct SQL/dashboard consumers now read JSON-encoded values (decode with from_json), existing tables are not migrated, and legacy string values that look like JSON atoms ("true", "1", "null") read back as typed values (True / 1 / None) — re-save affected rule sets after upgrading to normalize. The File/Volume (YAML/JSON) and Lakebase (JSONB) backends are unaffected. (#1319)

Full Changelog: v0.15.0...v0.16.0

Contributors

@mwojtyczka, @ghanse, @vb-dbrks, @SreeramaYeshwanthGowd, @mattfaltyn, @fedeflowers, @aarushisingh04, @IvannKurchenko, @abhyuday1203, @arnoN7, @AtomicGlance, @berrybluecode, @laurencewells, @neeraj-bhadani-08, @SaptarshiAcharyya99, @souravg-db2, @STEFANOVIVAS, @SyedIshmumAhnaf, @Vsatyam013