-
Notifications
You must be signed in to change notification settings - Fork 3
[Feature]: Extend verify to reconcile contract, dbt, and published lineage #31
Copy link
Copy link
Open
Description
Problem Statement
fluid verify already exists, but today it focuses on deployed drift such as schema, types, constraints, and location. There is not yet a minimal verification path that proves declared lineage, compiled batch lineage, and published lineage still agree.
Proposed Solution
- Reuse
fluid verifyinstead of adding a separate lineage command. - Add a lineage verification section that compares:
- declared lineage from
consumes[]andexposes[] - compiled lineage from dbt artifacts
- published lineage from marketplace or catalog APIs
- declared lineage from
- Keep v1 narrow to missing upstreams, missing outputs, and wrong output-to-model mapping.
- Treat dbt as enrichment and validation, not the lineage source of truth.
- Compare published lineage against the same normalized lineage model produced by the canonical lineage work.
Alternatives Considered
- Add a separate
fluid lineage verifycommand. - Make dbt the lineage source of truth.
Both approaches complicate the CLI surface or shift authorship away from FLUID contracts.
Area
CLI Commands / Verification / Provider behavior
How important is this to you?
Important — this extends an existing verification surface to cover a significant correctness gap.
Definition of Done
fluid verifycan detect lineage mismatches in addition to current drift checks- dbt-backed batch lineage can validate product-to-model expectations
- published lineage mismatches produce clear warnings or strict failures
- existing verify usage remains backward compatible
Non-goals
- no
fluid lineage verify - no OpenLineage runtime event ingestion yet
- no column-level lineage verification in v1
- no vendor-specific verification logic outside adapter boundaries
Additional Context
- Migrated from Trello: https://trello.com/c/V6ncsY8Y/42-fluid-extend-verify-to-reconcile-contract-dbt-and-published-lineage
- Source list:
Needs Triage - The Trello source references dbt artifacts in
businesses/telco/dbt/target/manifest.jsonandrun_results.jsonas the initial compiled-lineage inputs.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels