Skip to content

DataQ v1.1.0

Choose a tag to compare

@TheurgicDuke771 TheurgicDuke771 released this 22 Aug 01:42
· 654 commits to main since this release
53bd0b8

The post-v1 cycle: 2026-07-04 → 2026-08-21 (six planned weeks + a stretch week; ~620 commits, 14 new ADRs — 0028–0041, plus the ADR 0026 amendment). Tagged at the true cycle close — the stretch week alone added workspace RBAC, the MCP expansion, the security audit and the compliance track.

Highlights

The DQ platform leap

  • All monitor kinds real: comparison (cross-dataset reconciliation, ADR 0015), schema_drift, and anomaly (rolling z-score baseline with seasonality) join freshness/volume — every check classified on the DQ-dimension axis (ADR 0038) and rolled into per-asset scorecards.
  • Assets, lineage & incidents (ADR 0034/0037): the monitored table/file is a first-class entity — health rollup, OpenLineage-named identity, pulled lineage (dbt manifests, warehouse GET_LINEAGE, LineageProvider/Marquez), stateful deduped incidents with evidence cards, and the navigation inversion (assets lead the UI).
  • Scale-aware execution: sampling + partition batching + an OOM guardrail that refuses instead of SIGKILLing, sampled-ness recorded per result.

Reach

  • AWS as a second live reference deployment (ECS Fargate + RDS + ElastiCache + Cognito + Secrets Manager behind CloudFront) beside Azure, on one generic provider-neutral image + OIDC contract; IaC on OpenTofu.
  • Apache Iceberg as a fifth datasource (native pyiceberg, ADR 0030); S3-compatible stores (MinIO/R2/Wasabi/…); dbt as a third orchestration provider (ADR 0029); OpenBao / Vault-API secret store (ADR 0039); warehouse inventory sync (ADR 0040).

Access & AI

  • ⚠️ Workspace roles — Admin / Member / Viewer (ADR 0033). Connection mutations are now Admin-only. Breaking: promote connection-managing users to Admin before upgrading.
  • Email OTP sign-in (ADR 0032, the local/eval default), PATs (dq_live_, ADR 0026), rate limiting (ADR 0035).
  • MCP server 8 → 46 tools (23 read-only / 18 state-changing / 5 live-probe), with an honesty pass across all 46 — every tool states its blind spots as fields, not prose. No MCP tool is Admin-only; a credential never transits an LLM.

Compliance & security

  • Append-only audit trail (ADR 0041): every config mutation AND data read (results, comparison reports, profiles, dry-runs) recorded transactionally, admin-queryable, with whether regulated data was actually surfaced. Production-verified on both clouds.
  • Warehouse-tag PII classification (Snowflake + Unity Catalog) feeding the redaction ladder as a floor a suite policy cannot lift, plus an opt-in fail-closed mode; redaction follows the destination (interactive preview vs LLM/file/alert).
  • Security-surface audit of both deployments: self-registration closed (IdP + app-side allowlist), full browser security headers with enforcing CSP, CloudFront WAF + edge caching, non-root backend image, residency declaration + IaC region postcondition, compliance document set (sub-processors, DPIA input, breach runbook, DPA/BAA templates).

Full details: CHANGELOG.md · docs-site changelog · retro: docs/retro-v1.1.md

Full diff: v1.0.0...v1.1.0