-
Notifications
You must be signed in to change notification settings - Fork 3
[Feature]: Make consumes/exposes the canonical lineage source #30
Copy link
Copy link
Open
Description
Problem Statement
Native publish lineage is currently inconsistent across DPS and ODPS. In the current Entropy flow, products can publish successfully while upstream lineage is not persisted. The logic is split between datamesh_manager.py DPS input-port handling and odps.py input extraction, which makes lineage behavior drift-prone.
Proposed Solution
- Keep authored lineage in existing FLUID surfaces only:
consumes[],exposes[], andbuild. - Add one shared internal normalization step for lineage.
- Make both DPS and ODPS publisher paths consume the same normalized lineage model.
- Keep
fluid dmm publishunchanged at the CLI surface. - Treat
expects[]as an adapter projection only, not the primary authored lineage source.
Alternatives Considered
- Keep separate adapter-specific lineage logic in DPS and ODPS.
- Add a new top-level lineage command.
Both approaches increase surface area or preserve the current duplication.
Area
CLI Commands / Provider behavior / Contract lineage modeling
How important is this to you?
Critical — published lineage correctness is the core problem this issue is trying to solve.
Definition of Done
- native FLUID publish persists upstream lineage for Entropy ODPS orgs
- DPS and ODPS publishing derive lineage from the same internal source
- existing publish behavior for output ports remains intact
- regression coverage proves upstream lineage survives publish
Non-goals
- no new top-level CLI command
- no OpenLineage work in this issue
- no column-level lineage
- no new authored contract section
Additional Context
- Migrated from Trello: https://trello.com/c/nbYOb7Qz/41-fluid-make-consumes-exposes-the-canonical-lineage-source
- Source list:
Needs Triage
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels