Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-26560][StateBackends] Make the threshold of the overlap fraction of incremental restoring configurable #19106

Closed
wants to merge 1 commit into from
Closed

[FLINK-26560][StateBackends] Make the threshold of the overlap fraction of incremental restoring configurable #19106

wants to merge 1 commit into from

Conversation

fredia
Copy link
Contributor

@fredia fredia commented Mar 16, 2022

What is the purpose of the change

Currently, the threshold of the overlap fraction of incremental restoring OVERLAP_FRACTION_THRESHOLD is a hard-coded, fixed value.
OVERLAP_FRACTION_THRESHOLD is used to control how to restore a state handle, different thresholds can affect the performance of restoring. The behavior of deletion in restoring has been changed after FLINK-21321, the old threshold no longer fits the current situation.

To make it easier to modify the threshold according to different situations, changing OVERLAP_FRACTION_THRESHOLD to be configurable is suggested.

Brief change log

  • Add RESTORE_OVERLAP_FRACTION_THRESHOLD into RocksDBConfigurableOptions.

Verifying this change

This change added tests and can be verified as follows:

  • Add UT cases in RocksDBStateBackendConfigTest#testConfigurableOptionsFromConfig().

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented Mar 16, 2022 •
edited

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Myasuka
Myasuka requested changes Mar 16, 2022
View reviewed changes
Copy link
Member

@Myasuka Myasuka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for creating this PR. PTAL.

...rc/main/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtils.java Outdated Show resolved Hide resolved
...cksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBConfigurableOptions.java Outdated Show resolved Hide resolved
Myasuka
Myasuka reviewed Mar 16, 2022
View reviewed changes
Copy link
Member

@Myasuka Myasuka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update!
I think this PR looks good in general except a test case.

...est/java/org/apache/flink/contrib/streaming/state/RocksDBIncrementalCheckpointUtilsTest.java Outdated Show resolved Hide resolved
Myasuka
Myasuka approved these changes Mar 17, 2022
View reviewed changes
Copy link
Member

@Myasuka Myasuka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! LGTM.

@Myasuka
Copy link
Member

Myasuka commented Mar 17, 2022

@flinkbot run azure

2 similar comments
@Myasuka
Copy link
Member

Myasuka commented Mar 17, 2022

@flinkbot run azure

@fredia
Copy link
Contributor Author

fredia commented Mar 18, 2022

@flinkbot run azure

@fredia
Copy link
Contributor Author

fredia commented Mar 18, 2022 •
edited

CI passed at https://dev.azure.com/fredia/flink/_build/results?buildId=64&view=results.

@Myasuka Myasuka closed this in 38fe9c0 Mar 18, 2022
golden-yang pushed a commit to golden-yang/flink that referenced this pull request Mar 19, 2022
@fredia
[FLINK-26560][state] Make the threshold of the overlap fraction of in…
639e39c 
…cremental restoring configurable

This closes apache#19106.
JasonLeeCoding pushed a commit to JasonLeeCoding/flink that referenced this pull request May 27, 2022
@fredia @JasonLeeCoding
[FLINK-26560][state] Make the threshold of the overlap fraction of in…
825fb3b 
…cremental restoring configurable

This closes apache#19106.
zstraw pushed a commit to zstraw/flink that referenced this pull request Jul 4, 2022
@fredia
[FLINK-26560][state] Make the threshold of the overlap fraction of in…
57e89e1 
…cremental restoring configurable

This closes apache#19106.
@fredia fredia deleted the config_overlap branch April 18, 2023 07:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Myasuka Myasuka Myasuka approved these changes

Assignees
No one assigned
Labels
component=Runtime/StateBackends
Projects
None yet
Milestone
No milestone
4 participants
@fredia @flinkbot @Myasuka @rmetzger