Skip to content

[FLINK-37139][table-planner] Improve AdaptiveSkewedJoinOptimizationStrategy to make it effective in more scenarios#25989

Merged
JunRuiLee merged 1 commit into
apache:masterfrom
noorall:skewed-join-part4
Jan 16, 2025
Merged

[FLINK-37139][table-planner] Improve AdaptiveSkewedJoinOptimizationStrategy to make it effective in more scenarios#25989
JunRuiLee merged 1 commit into
apache:masterfrom
noorall:skewed-join-part4

Conversation

@noorall

@noorall noorall commented Jan 15, 2025

Copy link
Copy Markdown
Contributor

What is the purpose of the change

Currently, the scenarios in which the AdaptiveSkewedJoinOptimizationStrategy is effective are too limited. For example, when there is an output edge of the forward for unspecified type, the optimization cannot be applied. However, this optimization wouldn't break the data correctness of this kind output edges. Therefore, we will optimize this scenario to enable the optimization to be effective in more scenarios.

Brief change log

Improve AdaptiveSkewedJoinOptimizationStrategy to make it effective in more scenarios.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@noorall noorall marked this pull request as ready for review January 15, 2025 09:28

@zhuzhurk zhuzhurk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@flinkbot

flinkbot commented Jan 15, 2025

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

…rategy to make it effective in more scenarios

@JunRuiLee JunRuiLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JunRuiLee JunRuiLee merged commit d774cf1 into apache:master Jan 16, 2025
JunRuiLee added a commit to JunRuiLee/flink that referenced this pull request Jan 21, 2025
…zationStrategy to make it effective in more scenarios (apache#25989)"

This reverts commit d774cf1.
ryanvanhuuksloot pushed a commit to ryanvanhuuksloot/flink that referenced this pull request Feb 19, 2025
Au-Miner pushed a commit to Au-Miner/flink that referenced this pull request Feb 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants