Add Java SDK lineage IPC channel for OpenLineage #67111#67974
Open
haseebmalik18 wants to merge 1 commit into
Open
Add Java SDK lineage IPC channel for OpenLineage #67111#67974haseebmalik18 wants to merge 1 commit into
haseebmalik18 wants to merge 1 commit into
Conversation
haseebmalik18
requested review from
amoghrajesh,
ashb,
kaxil and
potiuk
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reserves the IPC channel and Java SDK API so JVM tasks can ship an opt-in
lineagepayload back to Python listeners.The supervisor adds an optional
lineagefield onSucceedTaskandTaskState, gated by aCadwynversion change so the wire stays compatible with older clients.BaseCoordinator.on_lineage_receivedre-emits the payload through the listener manager ason_lang_task_lineage_received, and the Python coordinator path leaves thelineage_handlerunwired so existing tasks pay nothing.On the Java side,
Context.setLineageandContext.putLineagegive users an opt-in builder. Tasks that never call them produce the same wire bytes as before, which the existing byte-for-byteCommTestassertion still proves.The OpenLineage provider listener does not yet consume
on_lang_task_lineage_received.closes: #67111