DataQ v1.0.0

DataQ v1.0.0

First stable release — a single-tenant data-quality monitoring platform built on Great Expectations (GX Core), delivered over an 8-week roadmap (2026-05-24 → 2026-07-03) and running in production on Azure Container Apps.

Highlights

Monitor 4 datasources. Write expectation checks against Snowflake, ADLS Gen2 flat files, AWS S3 flat files, and Unity Catalog (Databricks) — all through one ConnectionAdapter/CheckRunner seam, with credentials held in Azure Key Vault and never returned by the API.

React to your pipelines. Azure Data Factory and Apache Airflow integrate as orchestration providers (not datasources): near-real-time failure ingest via webhooks (shared-secret / HMAC-signed), trigger-on-success suite runs via provider-agnostic bindings, and a 10-minute polling fallback with gap recovery. A deliberate ADF failure was visible in DataQ 4m14s after firing.

Author checks like a product, not a config file. Catalog-driven check editor with per-expectation forms, Monaco-backed custom SQL checks (read-only, single-statement, SQL-datasources only — ADR 0019), column profiler on all four datasources, dry-run preview, severity tiers (warn/fail/critical banding with thresholds-as-policy — ADR 0016), and freshness & volume monitor kinds beyond row-level expectations (ADR 0012).

Run and watch. Async execution via Celery with live per-check progress, cancel, cron scheduling (DST-aware), run history with per-run outcome, a stuck-run reaper, and PII-minimising retention sweeps.

Know when it breaks. Teams, Slack, and email alerting behind a ResultPublisher seam with severity-aware routing, dedup, per-check snooze suppression, and per-suite notification config. Operationally-failed runs always alert.

See it. Monitoring dashboard (health score, trends, per-suite performance, period-over-period deltas), results explorer with drill-down to redacted failing-sample rows, and deep-linkable pages throughout.

Share it safely. Suite-scoped sharing (view/edit/owner) with workspace-admin governance (ADR 0027) — verified live: unshared suites 404 (no existence leak), edit-tier cannot delete, /admin is 403 for non-admins.

Talk to it. A FastMCP server exposes 8 curated tools at /mcp (same Azure AD token validation as REST, fail-closed) — verified end-to-end from a real VS Code MCP client config.

Deploy it anywhere (Azure today). One cloud-neutral frontend image with runtime-injected generic OIDC config (DATAQ_AUTH_*, no MSAL), the API on internal ingress behind the frontend proxy, in-repo Terraform, GHCR images, and a validated GitHub Actions deploy pipeline. AWS/GCP IaC is on the post-v1 roadmap.

Quality bar

• 1,289 backend tests, 98.4% line coverage (CI-enforced gate ≥80%, includes the API layer)
• 337 frontend tests, ~88% all-src line coverage (CI gate ≥80%) + 25 Playwright E2E specs in CI + an opt-in live-smoke lane
• Adversarial-input harness + mutation-testing spikes (mutmut 436/436 killed on the spiked service; Stryker 82.35%)
• Pre-tag QA: a hostile data-level workout against the running app (found and fixed the one blocker — NUL-byte input now a structured 422 everywhere), live cross-user authz verification as a non-admin user (15/15), and live webhook-auth hostility probes (7/7 structured 401s)
• 12 required CI checks: Ruff, Black, mypy (app+tests), pytest, frontend lint/format/test/E2E, secret scanning, Bandit, CodeQL, pip-audit/pnpm-audit CVE gates (zero ignored advisories)

Documentation

• User guides + reference: https://theurgicduke771.github.io/DataQ/ (getting started, concepts, datasources & checks, scheduling, orchestration, notifications, MCP setup, best practices, feature matrix, runbook)
• 27 Architecture Decision Records in docs/adr/
• Deployment guide with a complete environment-variable reference in deploy/

Known deferrals (decided, not forgotten)

• DataQ-issued API keys / service tokens — deferred to post-v1 with the shape confirmed (user-scoped PATs first; HTTP Basic auth rejected) — ADR 0026
• The live demo environment runs on Databricks Free Edition (non-commercial); it and all demo fixtures are slated for teardown before any commercial/marketplace use
• Full post-v1 backlog: 53 issues across 13 themes in context/post-v1-roadmap.md

Stats

405 commits · ~570 PRs+issues · 27 ADRs · 8 weeks · roadmap 187/189 (2 items consciously re-scoped to post-v1)