Phase 3: Discovery, catalog & trust (M1 + M2)

[kosminus]

Implements Phase 3 from planfull.md — the trust + discovery layer — scoped to M1 (certification & versioning) and M2 (catalog & lineage). Reuses the existing pgvector + keyword search (no tsvector) and uses sqlglot for lineage. Column profiling is deferred to a later milestone.

Milestone 1 — Certification & semantic versioning (migration 007)

• status / version / certified_by_id / certified_at on metrics, glossary, sample queries (+ cert stamps on saved queries).
• New SemanticVersion changelog table + versioning_service.py: one governed lifecycle (draft → in_review → certified → deprecated) for all four entity types. Editors submit-for-review/revert; admins certify/deprecate. Certifying validates SQL (read-only blocklist + sqlglot parse). Every content edit and transition appends a snapshot.
• POST .../{entity}/{id}/status + GET .../versions[/{v}] on the metric/glossary/sample-query/saved-query routers; PUTs bump version + snapshot. Saved-query status changes are routed through the governed lifecycle.

Milestone 2 — Catalog & lineage (migration 008)

• catalog_service.py: hybrid search across tables/columns/metrics/glossary/sample+saved queries/knowledge, reusing the existing pgvector embeddings + keyword scorer (no new full-text infra). Certified-first ranking; GET .../catalog/search + /facets.
• lineage_service.py + ArtifactDependency: sqlglot parses saved-query/metric SQL into table/column edges on create/update (best-effort, lazy import → graceful no-op if sqlglot absent). Per-artifact "what it touches" (.../{entity}/{id}/lineage) and impact view "what depends on this table" (.../catalog/lineage?table=).

Frontend

• Shared CertificationBadge / StatusActions / VersionHistory, wired into the Metrics, Glossary, and Saved Queries pages.
• New CatalogPage (search + facet sidebar + lineage detail drawer); nav item + route.

Tests & docs

• 22 new backend unit tests (177 total pass): versioning transitions/role-gating/cert-stamping, sqlglot extraction, catalog ranking.
• CHANGELOG, README, CLAUDE.md, and planfull.md status table updated. sqlglot added as the optional [lineage] extra and to the backend image.

Verification (live against the sample stack)

• Migrations 007/008 apply and reverse cleanly (downgrade 006 → upgrade head).
• Lifecycle: in_review → certified stamps owner/time; invalid transition → 422; changelog records both; revert clears the cert stamp.
• Catalog search ranks a certified metric ahead of higher-scored drafts.
• Lineage: a 2-table join produced 6 edges (all resolved to schema-cache IDs) + correct impact view.
• ruff clean on new files, new modules mypy-clean, npm run build + lint pass.

Notes

• One fix found in verification: status/PUT endpoints hit MissingGreenlet on the onupdate timestamp after an UPDATE (Postgres uses RETURNING on INSERT but not here) — resolved by refreshing the entity in versioning_service, same pattern as dashboard_service._finalize.
• Rebuild the backend image for lineage to populate in a running stack (sqlglot now in the Dockerfile).

Still open for Phase 3

• Column profiling (deferred — natural M3).
• Deep certify-time validation (missing columns / ambiguous joins), version diff-view UI, a SampleQueries page, and direct dashboard lineage edges.

🤖 Generated with Claude Code