Skip to content

[Feature]: Make consumes/exposes the canonical lineage source #30

@fas89

Description

@fas89

Problem Statement

Native publish lineage is currently inconsistent across DPS and ODPS. In the current Entropy flow, products can publish successfully while upstream lineage is not persisted. The logic is split between datamesh_manager.py DPS input-port handling and odps.py input extraction, which makes lineage behavior drift-prone.

Proposed Solution

  • Keep authored lineage in existing FLUID surfaces only: consumes[], exposes[], and build.
  • Add one shared internal normalization step for lineage.
  • Make both DPS and ODPS publisher paths consume the same normalized lineage model.
  • Keep fluid dmm publish unchanged at the CLI surface.
  • Treat expects[] as an adapter projection only, not the primary authored lineage source.

Alternatives Considered

  • Keep separate adapter-specific lineage logic in DPS and ODPS.
  • Add a new top-level lineage command.
    Both approaches increase surface area or preserve the current duplication.

Area

CLI Commands / Provider behavior / Contract lineage modeling

How important is this to you?

Critical — published lineage correctness is the core problem this issue is trying to solve.

Definition of Done

  • native FLUID publish persists upstream lineage for Entropy ODPS orgs
  • DPS and ODPS publishing derive lineage from the same internal source
  • existing publish behavior for output ports remains intact
  • regression coverage proves upstream lineage survives publish

Non-goals

  • no new top-level CLI command
  • no OpenLineage work in this issue
  • no column-level lineage
  • no new authored contract section

Additional Context

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions