Skip to content

Airbyte lineage not generated on public API (api/public/v1): source/destination config & connector type unresolved #30428

Description

@VolubleRain1

Affected module

Ingestion – Airbyte connector (pipeline lineage)

Describe the bug

When an Airbyte service uses the public API (apiVersion: api/public/v1), no pipeline→table lineage is generated even for the connector types the lineage generator supports (Postgres/MySQL/MSSQL/MongoDB). Ingestion reports success; lineage is silently empty.

Two causes in the source/destination resolution used by get_source_table_details / get_destination_table_details (ingestion/.../airbyte/utils.py):

  1. Config field name. AirbyteSourceResponse/AirbyteDestinationResponse only read sourceName/destinationName + connectionConfiguration. The public API returns the type under sourceType/destinationType and the config under configuration, so connectionConfiguration is Noneschema/database resolve to None → lineage skipped.
  2. Connector-type identifier. The support guard matches internal-API display names ("Postgres", "Microsoft SQL Server (MSSQL)"), but the public API returns slugs (postgres, mssql). Even after (1), the guard rejects the connector.

To Reproduce

Configure an Airbyte service with apiVersion: api/public/v1 and a Postgres→Postgres connection, enable lineage with dbServiceNames set, run ingestion → no lineage created.

Expected behavior

Lineage should resolve source/destination schema+database from whichever API shape responded, for all already-supported connector types.

Version

  • OpenMetadata: main / 1.13.x
  • Deployment: Docker

Related

Split from #26993 (that report additionally involves a Snowflake destination — warehouse-destination support is a separate enhancement).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions