[FLINK-31652][k8s] Handle the deleted event in case pod is deleted during the pending phase - #22311
Closed
SinBex wants to merge 2 commits into
Closed
[FLINK-31652][k8s] Handle the deleted event in case pod is deleted during the pending phase#22311SinBex wants to merge 2 commits into
SinBex wants to merge 2 commits into
Conversation
…ring the pending phase
Collaborator
Contributor
Author
|
@xintongsong Could you please help to review this PR in your free time? Thanks. |
huwh
reviewed
Mar 31, 2023
| import java.util.concurrent.CancellationException; | ||
| import java.util.concurrent.CompletableFuture; | ||
|
|
||
| import static io.fabric8.kubernetes.client.Watcher.Action.ADDED; |
Contributor
There was a problem hiding this comment.
It's better not use fabric classes in KubernetesResourceManagerDriver since these are Flink internal logic.
Contributor
Author
There was a problem hiding this comment.
Thanks for your suggestion. It is indeed better not to introduce an external class, I will use an internal enum to replace it.
| () -> { | ||
| for (KubernetesPod pod : pods) { | ||
| if (pod.isTerminated()) { | ||
| if (action == DELETED || pod.isTerminated()) { |
Contributor
There was a problem hiding this comment.
We need some annotations to explain why we need this logic.
…ted during the pending phase
Contributor
Author
|
Thank you so much for your review @huwh , I have updated the code, PTAL~ |
xintongsong
approved these changes
Apr 3, 2023
xintongsong
left a comment
xintongsong
left a comment
Contributor
There was a problem hiding this comment.
Thanks @SinBex for fixing this. The PR LGTM. I have only 1 minor comment, which I'll address myself while merging this.
| } | ||
|
|
||
| @Test | ||
| void testOnPodDeletedWithDeletedEvent() throws Exception { |
Contributor
There was a problem hiding this comment.
Suggested change
| void testOnPodDeletedWithDeletedEvent() throws Exception { | |
| void testOnPodDeletedBeforeScheduled() throws Exception { |
xintongsong
pushed a commit
that referenced
this pull request
Apr 3, 2023
…ring the pending phase This closes #22311
xintongsong
pushed a commit
that referenced
this pull request
Apr 3, 2023
…ring the pending phase This closes #22311
RocMarshal
pushed a commit
to RocMarshal/flink
that referenced
this pull request
May 9, 2024
…ring the pending phase This closes apache#22311
wfrong
pushed a commit
to tongcheng-elong/flink
that referenced
this pull request
Jan 8, 2026
…ring the pending phase This closes apache#22311
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Currently, in kubernetes deployment, if the taskmanager pod is deleted in 'Pending' phase, the flink job will get stuck and keep waiting for the pod scheduled. The cause reason is that the pod status will not be updated in time, so the KubernetesResourceManagerDriver won't detect the pod is terminated, but should handle the deleted event.
Brief change log
Verifying this change
KubernetesResourceManagerDriverTest#testOnPodDeletedWithDeletedEventDoes this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation