Skip to content

[Feature] Integration of dbt with Openlineage #11750

Open
@MassyB

Description

@MassyB

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

OpenLineage is an open-source project and standard designed to capture, collect, and visualize data lineage (the life-cycle and movement of data) across modern data ecosystems. It establishes a unified metadata model and an API for tools in the data pipeline ecosystem to report lineage and metadata in a standardized way.
In addition, OpenLineage captures data on pipeline runs which unlocks observability.

dbt should be fully integrated with openlineage like it's done now for Airflow.

Describe alternatives you've considered

N/A

Who will this benefit?

This will benefit users with the observability of their dbt pipelines. Openlineage (OL) is a standard. There are tools out there like Marquez that ingest and leverage those OL events.

It allows:

  1. lineage tracking of datasets
  2. observability of the pipeline executed
  3. observability of SQL queries executed in the data platform

Are you interested in contributing this feature?

Yes

Anything else?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions