[FLINK-36625] add lineage helper class for connector integration #25712

HuangZhenQiu · 2024-11-30T06:29:28Z

What is the purpose of the change

Add helper classes in stream java api for lineage integration in connectors.

Brief change log

Add TypeDatasetFacet class and Provide interface for easier extract type info from connectors.
Add LineageUtil to easily contract Lineage related POJOs

Verifying this change

This change added tests and can be verified as follows:

Added LineageUtilsTest to cover public functions

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: ( no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)
If yes, how is the feature documented? (not applicable)

flinkbot · 2024-11-30T06:33:58Z

CI report:

7771506 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

davidradl · 2024-12-02T08:45:38Z

flink-streaming-java/src/main/java/org/apache/flink/streaming/api/lineage/TypeDatasetFacet.java

+@PublicEvolving
+public interface TypeDatasetFacet extends LineageDatasetFacet {
+
+    TypeInformation getTypeInformation();


NIT: can we add @nonnull and @PublicEvolving to this method. I see examples in the Flink codebase where @PublicEvolving is on the methods as well as the class.

If it can be null it should be annotated with @nullable

Just to note, IMO @Nonnull usage adds a lot of bloat to the code (and Java is bloated enough on its own already) compared to its benefit. Using @Nullable is a lot more practical and reasonable when something can be null in my experience, but in this repo, none of the are (or should be) enforced.

davidradl · 2024-12-02T08:57:31Z

...ming-java/src/main/java/org/apache/flink/streaming/api/lineage/TypeDatasetFacetProvider.java

+     * Returns a type dataset facet or `Optional.empty` in case an implementing class is not able to
+     * resolve type.
+     */
+    Optional<TypeDatasetFacet> getTypeDatasetFacet();


I see this is a new interface but it is not referenced in the fix, what is the thinking here?

This interface is mainly for some connector in which return type is provided by internal class rather than the source/sink directly.

Thanks for the explanation. It would be good to understand when a connector author should use this interface. Could we add documentation around when this and the utility classes could/ should be used ?

davidradl · 2024-12-02T08:58:19Z

flink-streaming-java/src/main/java/org/apache/flink/streaming/api/lineage/LineageUtils.java

+        return datasetOf(name, namespace, Collections.singletonList(typeDatasetFacet));
+    }
+
+    public static LineageDataset datasetOf(


I wonder if these datasetOf constructor methods can be put in DefaultLineageDataset or a class that creates DefaultLineageDataset , maybe a DefaultLineageDatasetProvider

It is mainly to make the LineageDataset creation easier and simplify the code in each of connectors.

davidradl · 2024-12-05T08:42:21Z

Reviewed by Chi on 05/12/24. Asked submitter questions. Requests documentation be updated on how connector authors should / could use this helper class.

HuangZhenQiu · 2024-12-08T03:07:52Z

@davidradl
Given we have existing interfaces that haven't been documented, I plan to add en end to end native lineage page for this jira https://issues.apache.org/jira/browse/FLINK-35745. May I do it in a follow up PR?

HuangZhenQiu · 2024-12-08T06:21:29Z

After this PR is merged, I will update this PR #25762.

github-actions · 2025-04-09T06:25:58Z

This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/

If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.

github-actions · 2025-05-09T06:26:18Z

This PR has been closed since it has not had any activity in 120 days.
If you feel like this was a mistake, or you would like to continue working on it,
please feel free to re-open the PR and ask for a review.

ferenc-csaky · 2025-11-17T20:09:09Z

@flinkbot run azure

ferenc-csaky

LGTM, I'll merge this be tomorrow EOD if CI is green.

HuangZhenQiu · 2025-11-18T03:05:17Z

@ferenc-csaky
Do you want me to rebase main? BTW, we do need some help to review the lineage related PRs. It will be great, if you may help to review.

ferenc-csaky · 2025-11-18T04:29:22Z

@ferenc-csaky Do you want me to rebase main? BTW, we do need some help to review the lineage related PRs. It will be great, if you may help to review.

It's fine, these classes were not touched since then at all, no reason to rebase.

HuangZhenQiu force-pushed the FLINK-36625-lineage-helper branch from 8171f17 to 87538a2 Compare November 30, 2024 06:30

flinkbot added the component=API/DataStream label Nov 30, 2024

HuangZhenQiu force-pushed the FLINK-36625-lineage-helper branch from 87538a2 to edbdb1d Compare November 30, 2024 15:48

davidradl reviewed Dec 2, 2024

View reviewed changes

[FLINK-36625] add lineage helper class for connector integration

7771506

HuangZhenQiu force-pushed the FLINK-36625-lineage-helper branch from edbdb1d to 7771506 Compare December 4, 2024 06:03

github-actions bot added the stale label Apr 9, 2025

github-actions bot added the closed-stale label May 9, 2025

github-actions bot closed this May 9, 2025

ferenc-csaky mentioned this pull request Nov 17, 2025

[FLINK-34466][LINEAGE] Support dataset type facet for Avro apache/flink-connector-kafka#171

Open

ferenc-csaky reopened this Nov 17, 2025

ferenc-csaky removed stale closed-stale labels Nov 17, 2025

ferenc-csaky approved these changes Nov 17, 2025

View reviewed changes

github-actions bot added the community-reviewed PR has been reviewed by the community. label Nov 18, 2025

ferenc-csaky merged commit 1c2d953 into apache:master Nov 18, 2025

[FLINK-36625] add lineage helper class for connector integration #25712

[FLINK-36625] add lineage helper class for connector integration #25712

Uh oh!

Conversation

HuangZhenQiu commented Nov 30, 2024

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Nov 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

davidradl Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ferenc-csaky Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

davidradl Dec 2, 2024

Choose a reason for hiding this comment

Uh oh!

HuangZhenQiu Dec 3, 2024

Choose a reason for hiding this comment

Uh oh!

davidradl Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

davidradl Dec 2, 2024

Choose a reason for hiding this comment

Uh oh!

HuangZhenQiu Dec 3, 2024

Choose a reason for hiding this comment

Uh oh!

davidradl commented Dec 5, 2024

Uh oh!

HuangZhenQiu commented Dec 8, 2024

Uh oh!

HuangZhenQiu commented Dec 8, 2024

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

github-actions bot commented May 9, 2025

Uh oh!

ferenc-csaky commented Nov 17, 2025

Uh oh!

ferenc-csaky left a comment

Choose a reason for hiding this comment

Uh oh!

HuangZhenQiu commented Nov 18, 2025

Uh oh!

ferenc-csaky commented Nov 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Nov 30, 2024 •

edited

Loading

davidradl Dec 2, 2024 •

edited

Loading