Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FLINK-31077][runtime] Mark pending checkpoint onCompletionPromise complete only after the completed checkpoint is added to the store. #21943

Closed
wants to merge 3 commits into from
Closed

[FLINK-31077][runtime] Mark pending checkpoint onCompletionPromise complete only after the completed checkpoint is added to the store. #21943

wants to merge 3 commits into from

Conversation

JunRuiLee
Copy link
Contributor

What is the purpose of the change

Mark pending checkpoint onCompletionPromise complete only after the completed checkpoint is added to the store.

Brief change log

Mark pending checkpoint onCompletionPromise complete only after the completed checkpoint is added to the store.

Verifying this change

This change added tests and can be verified as follows: DefaultSchedulerTest#testTriggerCheckpointAndCompletedAfterStore

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot
Copy link
Collaborator

flinkbot commented Feb 15, 2023 •
edited

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

reswqa
reswqa reviewed Feb 15, 2023
View reviewed changes
Copy link
Member

@reswqa reswqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank @JunRuiLee for fixing this problem so quickly. I left some comments, please take a look~

flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerTest.java Outdated Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerTest.java Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerTest.java Outdated Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerTest.java Outdated Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerTest.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java Outdated Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/PendingCheckpointTest.java Outdated Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerTest.java Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerTest.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java Outdated Show resolved Hide resolved
@JunRuiLee
Copy link
Contributor Author

@reswqa Thanks for review and I've addressed all comments. PTAL!

reswqa
reswqa reviewed Feb 16, 2023
View reviewed changes
Copy link
Member

@reswqa reswqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JunRuiLee for addressing all comments, I only left minor comments, PTAL~

...a/org/apache/flink/runtime/scheduler/exceptionhistory/ExceptionHistoryEntryTestingUtils.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java Outdated Show resolved Hide resolved
@JunRuiLee JunRuiLee force-pushed the FLINK-31077 branch 3 times, most recently from fcb7a60 to 98dca84 Compare February 16, 2023 09:30
reswqa
reswqa approved these changes Feb 16, 2023
View reviewed changes
Copy link
Member

@reswqa reswqa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JunRuiLee for addressing all my comments, LGTM. For safety, we may need @zhuzhurk to confirm the changes about CheckpointCoordinator and PendingCheckpoint.

@zhuzhurk
Copy link
Contributor

@gaoyunhaii would you take a look at this change of checkpointing?

zhuzhurk
zhuzhurk reviewed Feb 20, 2023
View reviewed changes
Copy link
Contributor

@zhuzhurk zhuzhurk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for creating this PR!
I have a few comments. Please take a look.

flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java Outdated Show resolved Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java Show resolved Hide resolved
flink-runtime/src/test/java/org/apache/flink/runtime/scheduler/DefaultSchedulerTest.java Outdated Show resolved Hide resolved
@JunRuiLee
Copy link
Contributor Author

@zhuzhurk Thanks for CR and I've addressed all comments, PTAL!

zhuzhurk
zhuzhurk approved these changes Feb 21, 2023
View reviewed changes
Copy link
Contributor

@zhuzhurk zhuzhurk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@JunRuiLee
Copy link
Contributor Author

@gaoyunhaii Thanks for CR and I've addressed comment, PTAL!

gaoyunhaii
gaoyunhaii approved these changes Feb 23, 2023
View reviewed changes
Copy link
Contributor

@gaoyunhaii gaoyunhaii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @JunRuiLee for the fix, LGTM

@JunRuiLee
Copy link
Contributor Author

@flinkbot run azure

@zhuzhurk zhuzhurk closed this in eb17ec3 Feb 27, 2023
zhuzhurk pushed a commit that referenced this pull request Feb 27, 2023
@JunRuiLee @zhuzhurk
[FLINK-31077][runtime] Mark pending checkpoint onCompletionPromise co…
dca8195 
…mplete only after the completed checkpoint is added to the checkpoint store.

This closes #21943.
zhuzhurk pushed a commit that referenced this pull request Feb 27, 2023
@JunRuiLee @zhuzhurk
[FLINK-31077][runtime] Mark pending checkpoint onCompletionPromise co…
4c81591 
…mplete only after the completed checkpoint is added to the checkpoint store.

This closes #21943.
@JunRuiLee JunRuiLee deleted the FLINK-31077 branch August 31, 2023 04:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gaoyunhaii gaoyunhaii gaoyunhaii approved these changes

@zhuzhurk zhuzhurk zhuzhurk approved these changes

@reswqa reswqa reswqa approved these changes

Assignees
No one assigned
Labels
component=Runtime/Checkpointing
Projects
None yet
Milestone
No milestone
5 participants
@JunRuiLee @flinkbot @zhuzhurk @gaoyunhaii @reswqa