Repository navigation
DataQ v1.1.0
The post-v1 cycle: 2026-07-04 → 2026-08-21 (six planned weeks + a stretch week; ~620 commits, 14 new ADRs — 0028–0041, plus the ADR 0026 amendment). Tagged at the true cycle close — the stretch week alone added workspace RBAC, the MCP expansion, the security audit and the compliance track.
Highlights
The DQ platform leap
- All monitor kinds real:
comparison(cross-dataset reconciliation, ADR 0015),schema_drift, andanomaly(rolling z-score baseline with seasonality) join freshness/volume — every check classified on the DQ-dimension axis (ADR 0038) and rolled into per-asset scorecards. - Assets, lineage & incidents (ADR 0034/0037): the monitored table/file is a first-class entity — health rollup, OpenLineage-named identity, pulled lineage (dbt manifests, warehouse
GET_LINEAGE,LineageProvider/Marquez), stateful deduped incidents with evidence cards, and the navigation inversion (assets lead the UI). - Scale-aware execution: sampling + partition batching + an OOM guardrail that refuses instead of SIGKILLing, sampled-ness recorded per result.
Reach
- AWS as a second live reference deployment (ECS Fargate + RDS + ElastiCache + Cognito + Secrets Manager behind CloudFront) beside Azure, on one generic provider-neutral image + OIDC contract; IaC on OpenTofu.
- Apache Iceberg as a fifth datasource (native
pyiceberg, ADR 0030); S3-compatible stores (MinIO/R2/Wasabi/…); dbt as a third orchestration provider (ADR 0029); OpenBao / Vault-API secret store (ADR 0039); warehouse inventory sync (ADR 0040).
Access & AI
⚠️ Workspace roles — Admin / Member / Viewer (ADR 0033). Connection mutations are now Admin-only. Breaking: promote connection-managing users to Admin before upgrading.- Email OTP sign-in (ADR 0032, the local/eval default), PATs (
dq_live_, ADR 0026), rate limiting (ADR 0035). - MCP server 8 → 46 tools (23 read-only / 18 state-changing / 5 live-probe), with an honesty pass across all 46 — every tool states its blind spots as fields, not prose. No MCP tool is Admin-only; a credential never transits an LLM.
Compliance & security
- Append-only audit trail (ADR 0041): every config mutation AND data read (results, comparison reports, profiles, dry-runs) recorded transactionally, admin-queryable, with whether regulated data was actually surfaced. Production-verified on both clouds.
- Warehouse-tag PII classification (Snowflake + Unity Catalog) feeding the redaction ladder as a floor a suite policy cannot lift, plus an opt-in fail-closed mode; redaction follows the destination (interactive preview vs LLM/file/alert).
- Security-surface audit of both deployments: self-registration closed (IdP + app-side allowlist), full browser security headers with enforcing CSP, CloudFront WAF + edge caching, non-root backend image, residency declaration + IaC region postcondition, compliance document set (sub-processors, DPIA input, breach runbook, DPA/BAA templates).
Full details: CHANGELOG.md · docs-site changelog · retro: docs/retro-v1.1.md
Full diff: v1.0.0...v1.1.0