v0.14.0

What's Changed

• ML-based row-level anomaly detection (#990, #1055, #1062). DQX now offers ML-based row anomaly detection that automatically identifies unusual rows in data without requiring manually specified thresholds, enabling the detection of issues missed by rule-based checks. Users provide recent representative data, and DQX trains an Isolation Forest model that flags rows deviating from typical patterns at scoring time, with auto-discovery of relevant columns and segmentation where appropriate, plus per-row explanations of why a record was flagged. The feature integrates with MLflow for model registry, supports both training and scoring workflows, and complements existing rule-based and aggregate checks.
• DQX Studio app (Beta) — MVP release of DQX App (#1090) (#1040) (#1050) (#1034). DQX Studio is the no-code Databricks App for authoring and managing data quality rules through a browser-based UI. AI-assisted rule generation, in-app dry-run validation, scheduled rule execution with run history and per-check summary metrics, role-based access control (Admin, Approver, Author, Viewer, plus an orthogonal Runner role) backed by Databricks workspace groups, and a contextual AI assistant integrated into the UI. The bundle provisions all required resources automatically (app, SQL warehouse, task-runner job, schemas, volume) and exposes per-target variables for catalog, admin group, app name, warehouse name, and schema overrides. The app uses On-Behalf-Of (OBO) authentication so end users only see data they can access in Unity Catalog, and validates user-supplied checks with proper HTTP status codes (400 for malformed input). LLM configuration uses the calling user's OBO token on every request to ensure correct identity propagation in the deployed Apps environment.
• Added AI agent skills for DQX (#1125) (#1056). DQX now ships with Agent Skills under skills/ that teach AI assistants (Databricks Genie Code, Claude Code, Cursor, Copilot, and other tools following the open standard) how to use DQX correctly. The skills cover the public-API capabilities and are accompanied by an AGENTS.md canonical onboarding guide for AI coding agents, with a thin CLAUDE.md redirect for tools that look for it. A new docs guide documents installation and usage for each supported tool.
• Added has_no_aggr_outliers stateless rolling-window sigma outlier check (#1118). A new dataset-level quality check, has_no_aggr_outliers, has been introduced that detects outliers in time-series aggregates using a stateless rolling-window sigma method. The check is suitable for monitoring metrics such as daily transaction counts, hourly throughput, or any aggregate where deviations from a rolling baseline indicate quality issues, and complements the existing has_no_outliers MAD-based row-level check.
• Added are_polygons_mutually_disjoint geometry check function (#1061). A new geospatial check, are_polygons_mutually_disjoint, validates whether polygons in a column are mutually disjoint using ST_Intersects. The check supports row_filter, handles nulls and invalid geometries gracefully, and uses native Spark spatial intersections (rather than H3 indexing) for compatibility with Photon's spatial optimizations.
• Added null-safe support to foreign key check (#1106). The foreign_key check now accepts a null_safe parameter. By default, NULL values in the foreign key columns are ignored (SQL ANSI behavior). When null_safe=True, NULL foreign-key values are matched against NULL reference values. Note: enabling null_safe=True on a previously non-null-safe single-column FK changes the auto-generated rule name (a_not_exists_in_ref_b → struct_a_as_a_not_exists_in_ref_struct_b_as_a) and the violation message format.
• Added variable substitution support for check definitions (#1078). Check definitions now support {{ placeholder }} syntax for reusable templates, resolved at load time via a new variables parameter on load_checks() and load_checks_from_local_file(), or via default variables passed through ExtraParams at engine construction. The new resolve_variables() utility recursively replaces placeholders in all string fields of check definitions in a single pass and supports scalar types (str, int, float, bool, Decimal, datetime.date, datetime.datetime, datetime.time). Unresolved placeholders are logged as warnings.
• Added suppress_skipped option and skipped flag for skipped checks (#1063). A new suppress_skipped: bool = False option in ExtraParams allows checks skipped due to missing columns or invalid filters to produce no entry in _errors/_warnings and not cause rows to appear in the invalid DataFrame. Additionally, a new skipped boolean field has been added to dq_result_item_schema so skipped checks can be identified structurally without string-parsing the violation message.
• Added per-check-name breakdowns to summary metrics (#1097). The DQMetricsObserver now emits a new check_metrics row alongside the existing aggregates (input_row_count, error_row_count, warning_row_count, valid_row_count). The value is a JSON array of structs — one per check — with check_name, error_count, and warning_count, fitting the existing metric_name/metric_value schema without widening it. The change is backward compatible: existing metrics are unchanged and the new row is additive.
• Added versioning of checks with rule fingerprints (#1044). Checks now carry rule_fingerprint, rule_set_fingerprint, and created_at fields when saved to Delta or Lakebase storage, and rule_set_fingerprint is also stamped on summary metrics so every metric row can be traced back to the exact rule version that produced it. Each save creates a new versioned entry rather than overwriting prior history.
• Added partition and clustering support for output tables (#1012). The OutputConfig now accepts partition_by and cluster_by fields, allowing users to save DataFrames as partitioned or clustered tables. Liquid clustering is automatically applied the first time checks are saved to a liquid-clustered table, and the integration tests verify both partitioning and clustering behaviour end to end.
• Added configurable default criticality for profiler job (#1117). The profiler workflow now accepts a parameter to specify the default criticality (error or warn) for generated rules, allowing users to control rule severity at generation time rather than relying on a hardcoded default.
• Added schema validation rules generation from data contracts (#1043). The data contract rule generator now produces schema-validation rules from ODCS contracts (enabled via generate_schema_validation, defaulting to True), ensuring dataset schemas match contract definitions. A new InvalidPhysicalTypeError provides clearer error handling when physical types are missing or invalid in schema properties.
• Added end-to-end methods that load checks from storage (#1064). apply_checks_and_save_in_table and apply_checks_by_metadata_and_save_in_table now optionally load checks directly from a storage location (table or file), in addition to the existing option of using preloaded checks. Best-practice documentation has been updated with the recommended end-to-end patterns.
• Added solutions accelerators and industry demos (#1100). New industry-focused accelerators have been added under demos/dqx_demo_industry/: a Banking demo (dqx_banking_demo.py) focused on fraud detection and transaction monitoring, and a rebuilt Fashion demo (dqx_fashion_demo.py) with industry-specific custom check functions and 11 quality rules. The Manufacturing demo has been moved into the same subdirectory for consistency, and the demo documentation has been updated with a new "Industry Accelerators" section.
• Added intermediate demo for new users (#1041). A new intermediate demo has been added that can be presented in 5–10 minutes and showcases DQX's core functionality to someone seeing it for the first time.
• Added LLM-friendly documentation with llms.txt generation (#1029). The Docusaurus build now generates AI-accessible documentation in the standardized llms.txt format via the @signalwire/docusaurus-plugin-llms-txt plugin, with hierarchical organization so AI assistants and LLM-powered tools can consume DQX documentation more efficiently.
• Updated profiler implementation with rules-based profile builders (#1059). The DQX profiler has been refactored around a rules-based approach: profiles are now generated via registered profile builders, making it straightforward to add new profile types without modifying core profiler code.
• Improved summary stats report for string datatype columns (#1104). String columns no longer report min and max (lexicographic min/max is not meaningful for text data), and a count_distinct metric is now included for all column types in the profiler's summary stats output.
• Added py.typed marker for PEP 561 compliance (#1115). The package now ships with a py.typed marker file so downstream tools (mypy, pyright, etc.) recognise its existing type annotations instead of treating all databricks.labs.dqx imports as untyped.
• Support custom folder in CLI uninstall (#1066). The databricks labs uninstall dqx command now prompts for a custom workspace folder path (mirroring the install flow) and uses the new install_folder parameter on InstallationService.current() to locate installations outside the default /Users/<user>/.dqx location.
• Fixed AI-assisted sql_query generation and made has_valid_schema compatible with older Spark versions (#995). Resolved a subquery issue with sql_expression checks in Serverless v5 when the check name is auto-derived, made has_valid_schema compatible with Spark < 4, improved validation of required check function arguments, added agent guidelines, and added documentation on configuring DQX with Lakeflow Declarative Pipelines (LDP/DLT) for Materialized View incrementalization.
• Fixed has_valid_schema silently skipping validation for missing columns (#1042). A critical bug where has_valid_schema silently skipped validation for columns missing from the checked DataFrame has been fixed; missing columns are now reported as schema violations.
• Fixed save_results_in_table when DQObserver is present without metrics config (#1124). save_results_in_table now correctly handles the case where the calling DQEngine has an associated observer but no observation or metrics configuration is passed. The bundle has also been updated to use the direct deployment engine.
• Switched to WorkspaceClient for checking table existence in checks storage (#1035). TableChecksStorageHandler now uses WorkspaceClient to check for table existence when saving checks, replacing previous spark.catalog calls and improving compatibility across compute environments.
• Fixed Makefile coverage target to call the renamed test_coverage script (#1052).
• Migrated from hatch to uv package manager and hardened GitHub Actions workflows (#1098) (#1094) (#1096). The library has been migrated from hatch to uv for dependency and build management, GHA workflows have been refactored to increase infrastructure isolation and remove the Azure-login dependency, and performance benchmarks have been moved from per-PR runs to nightly. Dependency versions have been tightened, GitHub Actions are now pinned by SHA, and lock files have been cleaned up to remove registry-specific URLs and unused entries.
• Migrated test suite to native PySpark testing utilities (#1107). All tests now use pyspark.testing.utils.assertDataFrameEqual instead of chispa.assert_df_equality. The chispa test dependency has been removed, the centralized assert_df_equality_ignore_fingerprints wrapper has been updated to translate chispa-style kwargs (ignore_nullable, ignore_column_order, ignore_row_order) to PySpark equivalents, and chispa-specific transforms handling in the e2e PII notebook has been migrated to apply transforms before assertion.

BREAKING CHANGES!

• The default save mode for storing checks in Delta and Lakebase tables changed from overwrite to append. Rules are now versioned going forward — every save produces a new entry stamped with created_at, rule_set_fingerprint, and rule_fingerprint. To preserve the previous overwrite behaviour, explicitly pass mode="overwrite" when saving checks. (#1044)
• The DQ result item schema (the inner struct of _errors / _warnings array columns) gained two new nullable fields: rule_fingerprint and rule_set_fingerprint (#1044) and skipped (#1063). Pipelines that write to pre-existing Delta output tables created against the older schema will fail with a schema-mismatch error on the next write. Mitigation: pass {"mergeSchema": "true"} in OutputConfig.options (and similarly for the quarantine and metrics outputs) so Delta evolves the table on the first run after upgrade. Code that constructs dq_result_item_schema manually or asserts against an explicit StructType for _errors / _warnings must be updated to match the new shape.
• Parameter order changed in apply_checks_and_save_in_table and apply_checks_by_metadata_and_save_in_table. Update callers accordingly — see the methods' updated docstrings for the new signature. (#1064)

Contributors

@ghanse, @mwojtyczka, @vb-dbrks, @fedeflowers, @sundarshankar89, @berrybluecode, @laurencewells, @Roshan1299, @sheeluvikas, @cait-c, @balgaly, @moomindani, @Swayam-arora-2004, @STEFANOVIVAS, @IvannKurchenko, @vpottam-nvidia, @alexott

Full Changelog: v0.13.0...v0.14.0