Repository navigation
Oracle Data Integrator Pipelines #37458
|
I have a bunch of Oracle Data Integrator (ODI) pipelines I need to ingest into OpenMetadata to complete our lineage. I've built a few by interpreting the contents of the Package and Mapping XML files for a few ODI jobs and creating using the API. Has anyone ingested from ODI before? Am i going to need to create a custom connector for this? |
Replies: 1 comment
|
There's no ODI connector as of 2.0.7 (the latest release). I checked the pipeline service types and the ingestion sources at that tag, and
Either way, the piece that ties tables to ODI is from metadata.generated.schema.api.lineage.addLineage import AddLineageRequest
from metadata.generated.schema.type.entityLineage import ColumnLineage, EntitiesEdge, LineageDetails, Source as LineageSource
from metadata.generated.schema.type.entityReference import EntityReference
AddLineageRequest(edge=EntitiesEdge(
fromEntity=EntityReference(id=src_table.id, type="table"),
toEntity=EntityReference(id=tgt_table.id, type="table"),
lineageDetails=LineageDetails(
pipeline=EntityReference(id=odi_pipeline.id, type="pipeline"),
source=LineageSource.PipelineLineage,
columnsLineage=[ColumnLineage(fromColumns=[src_col_fqn], toColumn=tgt_col_fqn)],
),
))That's what the Fivetran connector does. OpenMetadata also accepts OpenLineage events at If the XML exports get painful, ODI's Java SDK reads mappings straight from the repository ( |
There's no ODI connector as of 2.0.7 (the latest release). I checked the pipeline service types and the ingestion sources at that tag, and
mainand 2.1.0-rc1 don't add one either. So yes, it's custom work, but what you've built from the Package/Mapping XML is the right foundation. Two ways to run it:Custom connector. Wrap your parser in a class extending
metadata.ingestion.api.steps.Sourcewhose_iteryieldsCreatePipelineRequests, thenAddLineageRequests. The defaultmetadata-restsink writes both. There's a working pipeline example. Package it, install it into your ingestion image, and add a Pipeline service of type Custom withsourcePythonClassset to your class (docs). You then get…