Kubelet: filter out terminated pods in SyncPods #7301
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This fixes #7250 Note that this PR handles most cases, but not all:
I think it's reasonable to assume that containers should only be garbage collected after a reasonable amount of time. As for the restart, we'd need a checkpoint of kubelet to handle that smoothly. One upside of this PR is that kubelet would stop wasting resources on terminated pods (examining containers, generating statuses and sending them to the apiserver). It would also avoid (potential) post-mortem status modifications. |
|
passed e2e |
|
/cc @dchen1107, @vmarmol |
| // Pod phase progresses monotonically. Once a pod has reach a final state, | ||
| // it should never leave irregardless of the restart policy. The statuses | ||
| // of such pods should not be changed, and there is no need to sync them. | ||
| pods := kl.FilterOutTerminatedPods(allPods) |
There was a problem hiding this comment.
Please also add TODO to explain those uncovered cases.
|
LGTM overall except one nit. |
|
LGTM |
yujuhong
force-pushed
the
no_resurrection
branch
from
f2870a7
to
cdeaac9
Compare
|
LGTM and will merge once it is green. |
dchen1107
added
the
lgtm
Once a pod reaches a terminated state (whether failed or succeeded), it should not transit out ever again. Currently, kubelet relies on examining the dead containers to verify that the container has already been run. This is fine in most cases, but if the dead containers were garbage collected, kubelet may falsely concluded that the pod has never been run. It would then try to restart all the containers. This change eliminates most of such possibilities by pre-filtering out the pods in the final states before sending updates to per-pod workers.
yujuhong
force-pushed
the
no_resurrection
branch
from
cdeaac9
to
b0e6926
Compare
vmarmol
added a commit
that referenced
this pull request
Apr 24, 2015
Kubelet: filter out terminated pods in SyncPods
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Once a pod reaches a terminated state (whether failed or succeeded), it should
not transit out ever again. Currently, kubelet relies on examining the dead
containers to verify that the container has already been run. This is fine
in most cases, but if the dead containers were garbage collected, kubelet may
falsely concluded that the pod has never been run. It would then try to restart
all the containers.
This change eliminates most of such possibilities by pre-filtering out the pods
in the final states before sending updates to per-pod workers.