v0.17.0

What's Changed

DQX Core

• Added the Lineage action (#1534). The new action CollectLineageAction collects upstream/downstream table and column lineage (including per-failed-column lineage) and last-modifier metadata for a run's input table and writes it to a configurable Delta table. It never fails the pipeline. Actions can now also pass a small extras payload to later actions in the same run, so for example an alert can reference the lineage table.
• Added support for ODCS type: library quality metrics in data contract rule generation (#1485). The rowCount, nullValues, missingValues, invalidValues, and duplicateValues metrics are mapped to DQX checks for all ODCS threshold fields (mustBe, mustNotBe, mustBeGreaterThan, mustBeBetween, etc.), carrying contract provenance and metric metadata. Malformed entries are warned about and skipped. A new process_library_rules parameter (default True) on generate_rules_from_contract lets you skip these entries.
• Added opt-in semantic-aware profiling to DQProfiler (#1491). A lightweight classification stage now runs between metric collection and check generation, so each column receives one consistent family of checks instead of contradictory overlaps (e.g. a low-cardinality column no longer gets both is_in and min_max). Pass a semantic_registry (e.g. SemanticRegistry.default()) to DQProfiler to enable it; built-in detectors classify columns as enum, key, measurement, or text, and the registry can be extended with custom detectors. Generated profiles record the detected semantic_type. Without a registry, profiling output is unchanged.
• Added the is_in_distribution dataset-level check (#1450). It compares the distribution of a categorical column against an expected value distribution and fails when the total variation distance exceeds the given distance.
• Added the is_valid_url row-level check (#1457). It validates values against the RFC 3986 absolute-URI grammar; a scheme is required, so relative references such as /path or example.com are rejected.
• Added the has_num_decimal_places row-level check (#1562). It validates that numeric values (including numbers stored as strings) have no more than the given number of decimal places.
• Added the is_geo_within_distance row-level geospatial check (#1510). It reports points farther than a maximum geodesic distance, in meters, from a reference point. distance accepts a number, a column, or a SQL expression; non-point geometries, unsupported SRIDs, and unparseable values are reported with dedicated messages. Geospatial documentation examples were fixed as well.
• Added geospatial profiling (#1535). Setting the new profile_geospatial profiler option generates geospatial checks for native GEOMETRY and GEOGRAPHY columns; geospatial_srid controls the SRID used to compute areas for GEOMETRY columns. Disabled by default.
• Added support for native GEOMETRY and GEOGRAPHY columns in geospatial checks (#1579). Set convert_column=False to apply the built-in geospatial checks to geospatial-typed columns. Values that cannot be parsed now fail with a message stating that the value is not a valid representation, and failure messages are consistent across the geospatial checks.
• Added UK and India formats to the is_valid_national_id check (#1461). The check now supports UK National Insurance numbers (GB) and Indian Permanent Account Numbers (IN).
• Added an allow_nulls option to comparison checks (#1492). is_equal_to, is_not_equal_to, is_not_less_than, is_not_greater_than, is_in_range, and is_not_in_range accept allow_nulls (default True). With allow_nulls=False, null values fail the check with a dedicated message instead of requiring a separate is_not_null rule. Default behavior is unchanged.
• Added a trim_strings option to is_unique (#1581). When enabled, leading and trailing spaces are trimmed from string keys before checking uniqueness; original values and messages are preserved. Default behavior is unchanged.
• Added checked_count and rule_fingerprint to per-check summary metrics (#1604). Each entry in check_metrics now records how many rows the check evaluated and the fingerprint of the rule, and the Query Results Cookbook shows how to use them. A row_filter passed in check function arguments is now validated like the check-level filter.
• Added an extensible registry for profiler column metrics (#1384). Custom column metrics can be registered with the new @register_profile_column_metric decorator, following the same pattern as register_rule and register_profile_builder. The built-in metrics moved to the registry with no behavior change.
• Streamlined execution of dataset-level checks (#1603). Aggregates are now computed from the input DataFrame and joined back with a single cross join, has_no_outliers and sql_query are evaluated lazily, and column expressions are validated once per run instead of once per check. This keeps query plans small with many dataset-level checks; runs with 16+ global aggregate checks that previously timed out now complete.
• Capped the supported Python version below 3.13 (#1503). DQX, DQX Studio, and the MCP server now declare requires-python with an upper bound of <3.13, matching the tested interpreters. The litellm and ruff pins were updated.
• Renamed the __about__.py module to __version__.py (#1475), following Python convention. Use from databricks.labs.dqx import __version__ to read the package version.
• Fixed escaping of column names during check execution (#1486). String column names that require SQL identifier escaping (e.g. names with spaces or non-ASCII letters) now resolve correctly in checks and generated rules, and auto-generated check names replace reserved characters with _.
• Fixed check-name escaping in check_metrics (#1471). Check names containing a single quote or backslash were reported as passing with an error_count of 0, and names containing a double quote produced unparseable JSON. Names are now escaped correctly.
• Fixed duplicate-key row multiplication in compare_datasets (#1488). Rows with duplicate matching keys are now paired one-to-one instead of producing a Cartesian product that multiplied output rows and reported false changes. Set raise_on_duplicate_keys=True to fail on duplicate keys instead.
• Fixed compare_datasets with MapType columns in the reference dataset (#1466). Shared columns are now excluded from the comparison when either the source or the reference declares them as MapType, as documented.
• Fixed DQEngine.save_checks to pass custom check functions to the storage handler (#1501). Checks using custom check functions can now be saved to Unity Catalog and Lakebase tables.
• Added a Customer 360 industry demo (#1472) using synthetic data.
• Added component-level code coverage reporting (#1480), including coverage for the MCP server.
• Made MCP server integration test coverage deterministic (#1494).
• Added the MCP server to the README key capabilities and made the documentation deploy script idempotent (#1468).

DQX Studio

• Made rules authored in DQX Studio loadable with DQX (#1531). A new dq_rules_core view exposes approved Studio rules in the DQX checks format, so pipelines can load them with DQEngine.load_checks and LakebaseChecksStorageConfig. Rule fingerprints are stored for every rule, the dq_quality_rules table was renamed to dq_resolved_rules, and a "Use outside the app" documentation section was added.
• DQX Studio: added full governance and authoring capabilities (#1428). This adds a Rules Registry with a draft → pending → approved lifecycle, versioning, and import/export; monitored tables and collections with schedules and approval queues; RBAC and object-level permissions; an equal-rule-weight DQ score with results breakdowns and quarantine samples; Ask Genie over results; a visual condition builder, custom and cross-table SQL rules, and AI rule suggestions; a Rules Marketplace of reusable rule packs; demo content; and a dedicated Studio documentation section.
• DQX Studio: added Databricks Marketplace installation with guided setup (#1500). Studio is packaged for Marketplace distribution, and a setup wizard runs live readiness checks with remediation guidance, provisioning the required schemas, volume, Lakebase, SQL warehouse, task-runner job, and migrations.
• DQX Studio: unified installation permissions for DAB and Marketplace (#1577). Both deployment paths now share one verified permission model. Studio derives its Unity Catalog storage from an existing catalog and a configurable prefix (default dqx_studio), collected by an admin setup form, and setup reports ready only when every required permission is verified.
• DQX Studio: versioned Studio releases independently of DQX (#1509). Studio uses its own version (starting at 0.1.0) and signed studio-vX.Y.Z release tags.
• DQX Studio: moved AI features to Unity Gateway model services (#1589). Legacy Foundation Model API endpoints (databricks-*) are being retired, so Studio now uses system.ai.<model> services (defaults system.ai.claude-sonnet-4-5 and system.ai.gte-large-en). Saved model names are migrated automatically and stored embeddings remain valid.
• DQX Studio: added group-by, a Schedules page, and improved contract import (#1557). Monitored tables and rules can be grouped by table or rule; a new Schedules page lists all schedules with health, next/last run, and admin pause/resume/delete. ODCS contract import moved to Tables and now creates generic per-column registry rules, and saving a schedule no longer fails on the scheduler grant.
• DQX Studio: added company branding and theme settings (#1580). Admins can set a company name, upload a logo, and choose or customize colour themes (with built-in presets) under Settings → Theme.
• DQX Studio: added external validation-status endpoints for uptime monitors (#1554). GET /monitoring/status/table/{table_fqn} and GET /monitoring/status/run/{run_id} return 200, 503, or 404 based on the latest or given run, so external monitors can poll Studio as a health check.
• DQX Studio: added resource tagging (#1546). Studio-owned resources (task-runner job, SQL warehouse, Lakebase project, Unity Catalog schemas, volumes, tables, and views, and demo resources) are tagged with app=dqx-studio, preserving existing tags.
• DQX Studio: added a PowerShell deployment command for Windows (#1560). make.ps1 app-deploy builds and deploys Studio on Windows, and a new release target deploys a prebuilt Studio release without building locally.
• DQX Studio: reduced SQL warehouse and Lakebase uptime (#1530, #1544, #1569). The scheduler backs off when no schedules are active and the Lakebase connection pool drains to zero, so Lakebase can scale to zero. Genie metadata is refreshed when a conversation starts instead of hourly, the temporary-view sweep runs daily, and background polls read run state from the Jobs API, so the SQL warehouse can auto-stop when Studio is idle. The default DAB warehouse auto-stop is now 10 minutes.
• DQX Studio: stored oversized run configs in a table instead of the wheels volume (#1547, #1564). The task runner no longer depends on psycopg.
• DQX Studio: improved setup (#1586, #1607). Non-administrators see who can finish setup, the task-runner step links directly to its job, and grants to users / account users satisfy every audience check. The AI step only confirms that endpoints exist. Grant statements can be run in one click as the signed-in user, the task-runner service principal setup is guided, and the catalog and user group are chosen from searchable pickers.
• DQX Studio: fixed Marketplace installation issues (#1561, #1567). Temporary and Genie schemas are derived from the bound schema and created during setup, required grants are applied or listed for an administrator, migration failures are reported instead of ignored, and task-runner and schema-level permissions are revalidated so setup no longer reports ready while runs or Genie would fail.
• DQX Studio: fixed bugs found in a bug bash (#1515). Fixes cover human-readable schedules for any day of the week, preserving the pass threshold in effect for each historical run, settings, reset, scheduled-run access, and many UI details.
• DQX Studio: fixed SQL warehouse error messages and Genie space provisioning (#1522). Warehouse change failures show actionable messages, and the managed Genie space is created under /Shared/dqx-studio, recovering from stale or inaccessible stored space IDs.
• DQX Studio: kept multi-column rules authorable (#1543). AI generation no longer picks a built-in check that cannot be bound to all the compared columns and falls back to the Condition Builder instead.
• DQX Studio: fixed schedule permission checks (#1590, #1591). Tables in the samples catalog no longer get stuck, permission failures are no longer reported as a slow warehouse, missing service principal grants are listed with the GRANT statements to run, and removing a schedule from a table's Schedule tab is saved immediately.
• DQX Studio: skipped scheduled runs while the previous run is still in progress (#1592), so overlapping schedules no longer interleave validation results.
• DQX Studio: fixed rule editor issues (#1582, #1583). Switching a saved SQL rule to a built-in check without negate now clears its polarity, and "Merge results back on" is pre-filled with {{slot}} placeholders so mapped columns resolve at run time.
• DQX Studio: fixed UI details (#1570, #1605). The profile row count is labelled "Rows profiled", the DQ score card is full width, and the button to remove a table from a collection is always visible.

BREAKING CHANGES!

• compare_datasets now rejects streaming DataFrames on either side with InvalidParameterError, because duplicate-key pairing requires bounded batch inputs. Compare bounded snapshots, or batch DataFrames within foreachBatch, instead. (#1488)
• The first parameter of DQMetricsObserver.get_metrics was renamed from check_names to checks (now list[DQRule] | list[str] | None), and df and skipped_checks parameters were added. Positional calls keep working, but keyword calls get_metrics(check_names=[...]) now raise TypeError; use get_metrics(checks=[...]). (#1604)
• A row_filter passed in check function arguments (e.g. for is_unique) that references a missing column now causes the check to be skipped and reported with skipped=True and a checked_count of 0, instead of failing with an AnalysisException. With suppress_skipped enabled, such a check leaves no entry in the results. (#1604)
• sql_query checks without merge_columns whose query returns more than one row now fail when the results are computed, instead of raising InvalidParameterError when apply_checks is called. (#1603)
• DQX Studio now requires Lakebase for application state; Delta-backed application state is no longer supported and has no automated migration path. When DQX_ADMIN_GROUP is unset, members of the workspace admins group get the in-app ADMIN role. (#1500)
• Existing DQX Studio installations (DAB or Marketplace) must be removed and reinstalled because of the new permission and storage model; there is no upgrade path. See app/DEPLOYMENT.md → "Removing a previous installation". (#1577)

Full Changelog: v0.16.0...v0.17.0

Contributors: @mwojtyczka, @alvaromoureupm, @amising6, @AtomicGlance, @berrybluecode, @Christopher-Lawford, @fedeflowers, @ghanse, @GewoonMaarten, @IvannKurchenko, @keerthimath, @laurencewells, @mattfaltyn, @moomindani, @OGordon100, @simplegaurav, @vb-dbrks