Skip to content

[Feature]: Extend verify to reconcile contract, dbt, and published lineage #31

@fas89

Description

@fas89

Problem Statement

fluid verify already exists, but today it focuses on deployed drift such as schema, types, constraints, and location. There is not yet a minimal verification path that proves declared lineage, compiled batch lineage, and published lineage still agree.

Proposed Solution

  • Reuse fluid verify instead of adding a separate lineage command.
  • Add a lineage verification section that compares:
    • declared lineage from consumes[] and exposes[]
    • compiled lineage from dbt artifacts
    • published lineage from marketplace or catalog APIs
  • Keep v1 narrow to missing upstreams, missing outputs, and wrong output-to-model mapping.
  • Treat dbt as enrichment and validation, not the lineage source of truth.
  • Compare published lineage against the same normalized lineage model produced by the canonical lineage work.

Alternatives Considered

  • Add a separate fluid lineage verify command.
  • Make dbt the lineage source of truth.
    Both approaches complicate the CLI surface or shift authorship away from FLUID contracts.

Area

CLI Commands / Verification / Provider behavior

How important is this to you?

Important — this extends an existing verification surface to cover a significant correctness gap.

Definition of Done

  • fluid verify can detect lineage mismatches in addition to current drift checks
  • dbt-backed batch lineage can validate product-to-model expectations
  • published lineage mismatches produce clear warnings or strict failures
  • existing verify usage remains backward compatible

Non-goals

  • no fluid lineage verify
  • no OpenLineage runtime event ingestion yet
  • no column-level lineage verification in v1
  • no vendor-specific verification logic outside adapter boundaries

Additional Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions