DataQ v1.2.0

The DQ-intelligence and operability cycle (2026-08-22 → 2026-10-04). Full entries are on the changelog page; headlines:

Added

• Six SQL datasources on one engine-generic base (ADR 0045): PostgreSQL, MySQL / MariaDB,
  Trino, Amazon Athena and Amazon Redshift, plus SQL Server / Azure SQL / Synapse / Fabric SQL
  (ADR 0044). ADLS Gen2 connections can sign in as a service principal, which also covers
  Fabric OneLake.
• Per-check engines (ADR 0036): Snowflake DMF, including custom DMFs, and Databricks DQX
  run in the user's own workspace, beside Great Expectations.
• LLM assistance, off by default (ADR 0042): SQL generation, check suggestions and
  root-cause narratives through a provider seam (Anthropic, any OpenAI-compatible endpoint,
  Snowflake Cortex), each validated before anything is saved.
• Automated coverage (ADR 0047): system-owned suites over the asset inventory, a review
  queue of suggested rules, and coverage and false-positive figures on the Dashboard.
• Suites as code: YAML suite files, validate, drift and apply, and a generated Python
  client with the dataq CLI, attached to each release, under an API compatibility policy.
• A pipeline gate (ADR 0046): POST /orchestration/gate lets a pipeline stage start its
  bound suites and wait for the verdict.
• Operating a workspace: reusable notification channels; the admin area as six pages
  with in-app membership (ADR 0043), guided offboarding and a workspace-health view; a hash
  chain over the audit log.
• Scores and bulk work: health scores for assets, connections and dimensions; bulk snooze,
  delete and thresholds for checks; a first-run path and empty states that say what to do next.
• MCP: 46 → 54 tools, including column lineage tracing, notification channels, curated
  docs, coverage figures and suggested rules.
• A production-like local stack: docker-compose.ghcr.yml serves only the UI, over HTTPS
  from a CA it generates, with TLS on every internal hop and data and a persistent vault in a
  directory. A --profile demo sample warehouse shows real results on first sign-in.
• Accessibility: every route passes the automated axe checks in both themes.
• Docs: rebuilt on a task-first layout, versioned per release, with an upgrade guide and
  a support policy.

Changed

• AWS stack — Slack alerts are off unless slack_webhook_secret_name is set (#2292). The
  stack used to hardcode SLACK_WEBHOOK_SECRET_NAME=channel-slack-webhook without creating that
  secret. An AWS deployment that created <prefix>/channel-slack-webhook itself must set
  slack_webhook_secret_name = "channel-slack-webhook" before its next task-definition replace,
  or Slack alerts stop without a log line.

• Breaking — connections are tested before they are saved (#1927). POST /connections,
  a PATCH that changes config or a credential, and POST /connections/{id}/reauth refuse a
  config that fails its test with 422 connection_test_failed_on_save and write nothing (a
  re-auth keeps the old credential). An Admin can send skip_test: true to save anyway; the
  audit event records it. PUT /admin/llm tests an enabled provider first
  (422 llm_test_failed_on_save).

• Breaking — Unity Catalog DATE columns on the frame lane (sampled suites,
  UC_SQL_PUSHDOWN=false, and the frame-only types such as to_be_of_type) are now
  evaluated as dates, as on Parquet and Iceberg. Before, a date bound errored and a set
  of ISO dates failed every row. A to_be_of_type check on a UC DATE column that
  names datetime64[ns] / datetime64[s] must be changed to type_: date. Datetime
  bounds at midnight keep working; a bound with a time of day on a DATE column errors.

• Breaking — the prebuilt-image stack publishes only the UI, over HTTPS, and keeps its data
  in a directory (DATAQ_DATA_DIR, default ./dataq-data). Take the compose file and the
  images from the same release.

• This release adds 33 database migrations. They run before the application starts. Back
  up first; see the upgrade guide for rolling back.

Security

• The custom-SQL guard lexes each engine's own quoting.
• Sensitive columns stay masked in incident evidence, alerts and LLM prompts.

Upgrading

Back up first, then follow the upgrade guide. The Python client and the dataq CLI are attached below (pip install dataq_client-1.2.0-py3-none-any.whl).

Full changelog: v1.1.0...v1.2.0

DataQ v1.1.0

The post-v1 cycle: 2026-07-04 → 2026-08-21 (six planned weeks + a stretch week; ~620 commits, 14 new ADRs — 0028–0041, plus the ADR 0026 amendment). Tagged at the true cycle close — the stretch week alone added workspace RBAC, the MCP expansion, the security audit and the compliance track.

Highlights

The DQ platform leap

• All monitor kinds real: comparison (cross-dataset reconciliation, ADR 0015), schema_drift, and anomaly (rolling z-score baseline with seasonality) join freshness/volume — every check classified on the DQ-dimension axis (ADR 0038) and rolled into per-asset scorecards.
• Assets, lineage & incidents (ADR 0034/0037): the monitored table/file is a first-class entity — health rollup, OpenLineage-named identity, pulled lineage (dbt manifests, warehouse GET_LINEAGE, LineageProvider/Marquez), stateful deduped incidents with evidence cards, and the navigation inversion (assets lead the UI).
• Scale-aware execution: sampling + partition batching + an OOM guardrail that refuses instead of SIGKILLing, sampled-ness recorded per result.

Reach

• AWS as a second live reference deployment (ECS Fargate + RDS + ElastiCache + Cognito + Secrets Manager behind CloudFront) beside Azure, on one generic provider-neutral image + OIDC contract; IaC on OpenTofu.
• Apache Iceberg as a fifth datasource (native pyiceberg, ADR 0030); S3-compatible stores (MinIO/R2/Wasabi/…); dbt as a third orchestration provider (ADR 0029); OpenBao / Vault-API secret store (ADR 0039); warehouse inventory sync (ADR 0040).

Access & AI

• ⚠️ Workspace roles — Admin / Member / Viewer (ADR 0033). Connection mutations are now Admin-only. Breaking: promote connection-managing users to Admin before upgrading.
• Email OTP sign-in (ADR 0032, the local/eval default), PATs (dq_live_, ADR 0026), rate limiting (ADR 0035).
• MCP server 8 → 46 tools (23 read-only / 18 state-changing / 5 live-probe), with an honesty pass across all 46 — every tool states its blind spots as fields, not prose. No MCP tool is Admin-only; a credential never transits an LLM.

Compliance & security

• Append-only audit trail (ADR 0041): every config mutation AND data read (results, comparison reports, profiles, dry-runs) recorded transactionally, admin-queryable, with whether regulated data was actually surfaced. Production-verified on both clouds.
• Warehouse-tag PII classification (Snowflake + Unity Catalog) feeding the redaction ladder as a floor a suite policy cannot lift, plus an opt-in fail-closed mode; redaction follows the destination (interactive preview vs LLM/file/alert).
• Security-surface audit of both deployments: self-registration closed (IdP + app-side allowlist), full browser security headers with enforcing CSP, CloudFront WAF + edge caching, non-root backend image, residency declaration + IaC region postcondition, compliance document set (sub-processors, DPIA input, breach runbook, DPA/BAA templates).

Full details: CHANGELOG.md · docs-site changelog · retro: docs/retro-v1.1.md

Full diff: v1.0.0...v1.1.0

DataQ v1.0.0

DataQ v1.0.0

First stable release — a single-tenant data-quality monitoring platform built on Great Expectations (GX Core), delivered over an 8-week roadmap (2026-05-24 → 2026-07-03) and running in production on Azure Container Apps.

Highlights

Monitor 4 datasources. Write expectation checks against Snowflake, ADLS Gen2 flat files, AWS S3 flat files, and Unity Catalog (Databricks) — all through one ConnectionAdapter/CheckRunner seam, with credentials held in Azure Key Vault and never returned by the API.

React to your pipelines. Azure Data Factory and Apache Airflow integrate as orchestration providers (not datasources): near-real-time failure ingest via webhooks (shared-secret / HMAC-signed), trigger-on-success suite runs via provider-agnostic bindings, and a 10-minute polling fallback with gap recovery. A deliberate ADF failure was visible in DataQ 4m14s after firing.

Author checks like a product, not a config file. Catalog-driven check editor with per-expectation forms, Monaco-backed custom SQL checks (read-only, single-statement, SQL-datasources only — ADR 0019), column profiler on all four datasources, dry-run preview, severity tiers (warn/fail/critical banding with thresholds-as-policy — ADR 0016), and freshness & volume monitor kinds beyond row-level expectations (ADR 0012).

Run and watch. Async execution via Celery with live per-check progress, cancel, cron scheduling (DST-aware), run history with per-run outcome, a stuck-run reaper, and PII-minimising retention sweeps.

Know when it breaks. Teams, Slack, and email alerting behind a ResultPublisher seam with severity-aware routing, dedup, per-check snooze suppression, and per-suite notification config. Operationally-failed runs always alert.

See it. Monitoring dashboard (health score, trends, per-suite performance, period-over-period deltas), results explorer with drill-down to redacted failing-sample rows, and deep-linkable pages throughout.

Share it safely. Suite-scoped sharing (view/edit/owner) with workspace-admin governance (ADR 0027) — verified live: unshared suites 404 (no existence leak), edit-tier cannot delete, /admin is 403 for non-admins.

Talk to it. A FastMCP server exposes 8 curated tools at /mcp (same Azure AD token validation as REST, fail-closed) — verified end-to-end from a real VS Code MCP client config.

Deploy it anywhere (Azure today). One cloud-neutral frontend image with runtime-injected generic OIDC config (DATAQ_AUTH_*, no MSAL), the API on internal ingress behind the frontend proxy, in-repo Terraform, GHCR images, and a validated GitHub Actions deploy pipeline. AWS/GCP IaC is on the post-v1 roadmap.

Quality bar

• 1,289 backend tests, 98.4% line coverage (CI-enforced gate ≥80%, includes the API layer)
• 337 frontend tests, ~88% all-src line coverage (CI gate ≥80%) + 25 Playwright E2E specs in CI + an opt-in live-smoke lane
• Adversarial-input harness + mutation-testing spikes (mutmut 436/436 killed on the spiked service; Stryker 82.35%)
• Pre-tag QA: a hostile data-level workout against the running app (found and fixed the one blocker — NUL-byte input now a structured 422 everywhere), live cross-user authz verification as a non-admin user (15/15), and live webhook-auth hostility probes (7/7 structured 401s)
• 12 required CI checks: Ruff, Black, mypy (app+tests), pytest, frontend lint/format/test/E2E, secret scanning, Bandit, CodeQL, pip-audit/pnpm-audit CVE gates (zero ignored advisories)

Documentation

• User guides + reference: https://theurgicduke771.github.io/DataQ/ (getting started, concepts, datasources & checks, scheduling, orchestration, notifications, MCP setup, best practices, feature matrix, runbook)
• 27 Architecture Decision Records in docs/adr/
• Deployment guide with a complete environment-variable reference in deploy/

Known deferrals (decided, not forgotten)

• DataQ-issued API keys / service tokens — deferred to post-v1 with the shape confirmed (user-scoped PATs first; HTTP Basic auth rejected) — ADR 0026
• The live demo environment runs on Databricks Free Edition (non-commercial); it and all demo fixtures are slated for teardown before any commercial/marketplace use
• Full post-v1 backlog: 53 issues across 13 themes in context/post-v1-roadmap.md

Stats

405 commits · ~570 PRs+issues · 27 ADRs · 8 weeks · roadmap 187/189 (2 items consciously re-scoped to post-v1)