Data lineage for Kedro

[ravi-kumar-pilla]

Summary

Session 1 of the Data lineage tech design is ready for team feedback.

This is the first of a three part series. Session 1 sets vision, scope, and direction. We will only write Sessions 2 and 3 once we align here.

Please read Session 1 and comment on the design decisions below. Right now we care about alignment on direction, not implementation detail.

---

Background

• Parent issue: #5311: Data quality management for Kedro pipelines (lineage is one workstream within this)
• Tech design doc (Session 1): docs/proposals/data-lineage-spike.md
• Core vs plugin: docs/proposals/data-lineage-core-vs-plugin.md
• Spike experiment guide: docs/proposals/kedro-openlineage-marquez-local-setup.md
• Prior exploration: Discussion #4054: OpenLineage exploration

---

What Session 1 proposes

Kedro already has a dependency graph (node inputs/outputs) and a local Workflow view. That works well for local debugging, but not for run history, lineage across the org, or compliance. Here is the proposal:

1. Static graph from kedro.inspection / get_project_snapshot() (already in progress)
2. Runtime events in OpenLineage format, emitted by an official kedro-openlineage plugin
3. Kedro-Viz for local dev (flowchart plus the most recent Workflow view)
4. Marquez / DataHub when teams need an enterprise catalog, audit trail, and lineage across tools

We are not building a full enterprise catalog inside Kedro.

---

Design decisions we need alignment on

Please comment on each decision. 👍 / 👎 reactions are welcome as a rough signal, but threaded comments with your reasoning are what help us most.

Decision A: Scope (extend Viz, don't replace it)

Proposal: Keep Kedro-Viz Flowchart and Workflow view for local dev. Add OpenLineage so production runs can reach Marquez/DataHub. Do not try to turn Kedro-Viz into an enterprise catalog.

Question: Do you agree with this scope boundary?

---

Decision B: Runtime events use OpenLineage, not custom JSON

Proposal: Replace the custom .viz/kedro_pipeline_events.json format with standard OpenLineage RunEvents. DataHub and Marquez already ingest this format.

Question: Is OpenLineage the right standard for Kedro runtime lineage events?

Alternatives to flag: Keep a Kedro specific event schema. Or support OpenLineage only as an optional export.

---

Decision C: Emission lives in a plugin, not Kedro core or Kedro-Viz

Proposal: Build on the community kedro-openlineage PoC as an official kedro-org plugin. Hooks listen during kedro run and emit events. This works in CI and production without installing Kedro-Viz.

Question: Do you agree runtime OpenLineage emission should be a plugin, not core?

Alternatives to flag: Ship OpenLineage emission in Kedro core (Airflow style). Or keep it community maintained only.

See core vs plugin rationale for the comparison with dbt, Dagster, and Airflow.

---

Decision D: One emitter, two outputs

Proposal: A single OpenLineage emitter with two configurable sinks:

Output	Destination	Purpose
File	.viz/lineage_events.json	Kedro-Viz Workflow view (local dev, last run only)
HTTP	Marquez or DataHub	Production and org wide run history

Question: Do you agree we should consolidate today's separate hook paths into one emitter, two outputs?

Note: The file sink is overwritten each run (last run only). The HTTP sink appends to the catalog server's history. That history lives in the org's data catalog (Marquez, DataHub, or similar).

---

Decision E: Split metadata.kedro vs metadata.kedro-viz in catalog.yml

Proposal: Governance fields (owner, description, consumers) live under metadata.kedro and can be published to enterprise catalogs. UI only fields (layer, preview, styling) stay under metadata.kedro-viz.

model_input_table:
  type: pandas.ParquetDataset
  filepath: data/03_primary/model_input.parquet
  metadata:
    kedro:                    # shared: lineage, validation, enterprise catalog
      owner: data-team@company.com
      description: "Features fed into training"
    kedro-viz:                # UI only: not published externally
      layer: primary

Question: Do you agree with this namespace split before we define the full schema in Session 2?

---

Decision F: Validation on the same lineage event stream

Proposal: Validation results attach to OpenLineage RunEvents via the dataQualityAssertions facet. No separate validation event stream.

Question: Do you agree lineage and validation should share one event stream (aligned with #5311)?

---

Roadmap alignment (high level)

If the decisions above look right, this is the phased plan Sessions 2 and 3 will detail:

Phase	Goal	Primary owner
0	Kedro-Viz uses get_project_snapshot() (in progress)	Kedro core + Kedro-Viz
1	Define metadata.kedro schema, extend snapshot, document table level lineage	Docs, core, Viz
2	OpenLineage emitter (file + HTTP), validation facets, Workflow view migration	kedro-openlineage plugin
3	Column lineage (pandas + Spark)	Plugin + Viz
4	Publish structure and metadata to enterprise catalog, impact analysis	Plugin

Question: Does this phase ordering make sense for what Kedro users need first?

---

Explicitly out of scope for Session 1

These are Session 2+ topics. Please don't block Session 1 alignment on them. We will design them next:

• OpenLineage job and dataset naming convention (project scoped vs storage based, MemoryDataset, transcoding, factories, versioning)
• One runId per kedro run (UUID mapping from Kedro session ID)
• FAIL and ABORT events and best effort emission when sinks are down
• ThreadRunner and ParallelRunner behaviour
• Migration path from PipelineRunStatusHook and DatasetStatsHook

---

How to give feedback

1. Read Session 1 in the spike doc (sections 1.1 to 1.5). The companion docs are on the same branch: core vs plugin and Marquez local setup.
2. Comment on the decisions above. If you can, reply in a thread per decision (A through F) so we can see what is settled.
   • Most important: B (OpenLineage), C (plugin vs core), D (one emitter, two outputs), E (metadata.kedro split), F (validation on the same stream).
   • Also welcome: A (scope boundary) and your take on the roadmap phase ordering.
3. Optional: Try Experiment 1 (kedro-openlineage to Marquez) and share what you learned.
4. React with 👍 if you broadly agree with Session 1 direction, 👎 if you don't. Either way, a short comment explaining why would really help.

---

What happens next

• Feedback window: ~ 2 weeks from when this discussion opens.
• During the window: Comments here will drive updates on the chore/spike_lineage branch. We will revise Session 1 in the spike doc if needed and reply in thread when a decision looks settled.
• If we align: Merge the spike docs, then write Session 2 (Phases 0 to 2 implementation design) in the same doc. We can follow up with another discussion or a live tech design session.
• If we don't align: Revise Session 1 from the comments, update the branch, and bring the open decisions back here before moving on.
• For later: Column lineage and enterprise catalog publishing stay in Session 3 (Phases 3 to 4). We won't design those until Sessions 1 & 2 are agreed on.

Thanks for reading. Looking forward to your thoughts.

[Huongg]

hey @ravi-kumar-pilla thank you for the TD and the summary in discussion here.

For me to vote on these, I think I'd need a rough engineering effort for each option.

I currently lean towards decision A because it looks simplest, but that could be a wrong assumption, maybe from the work you already did, you might say the other decision is actually less effort

Even just a rough S/M/L would help me understand the trade-offs better.

And would love to know which one is your recommendation?