Skip to content

Comments

[FLINK-37082][table-planner] Disable adaptive join optimization when enable batch job progress recovery.#25931

Merged
zhuzhurk merged 1 commit intoapache:masterfrom
SinBex:JIRA-37082
Jan 9, 2025
Merged

[FLINK-37082][table-planner] Disable adaptive join optimization when enable batch job progress recovery.#25931
zhuzhurk merged 1 commit intoapache:masterfrom
SinBex:JIRA-37082

Conversation

@SinBex
Copy link
Contributor

@SinBex SinBex commented Jan 9, 2025

What is the purpose of the change

Enabling Batch job progress recovery and adaptive batch execution to work together will be supported in future versions.

To prevent issues with job recovery when users configure Batch job progress recovery, we will temporarily disable adaptive join optimization in scenarios where Batch job progress recovery is enabled.

Brief change log

  • Disable adaptive join optimization when enable batch job progress recovery.

Verifying this change

This change added tests and can be verified as follows:

  • Test with ut AdaptiveJoinTest::testAdaptiveJoinWithBatchJobRecovery.
  • Enable the config option execution.batch.job-recovery.enabled and check whether adaptive join optimization is disabled.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: ( no )
  • The runtime per-record code paths (performance sensitive): (no )
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no )

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not documented)

Copy link
Contributor

@zhuzhurk zhuzhurk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@flinkbot
Copy link
Collaborator

flinkbot commented Jan 9, 2025

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@zhuzhurk zhuzhurk merged commit 0975022 into apache:master Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants