Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure that Reason and Message are preserved on pod status #103785

Merged
merged 3 commits into from Jul 20, 2021

Conversation

smarterclayton
Copy link
Contributor

@smarterclayton smarterclayton commented Jul 19, 2021

The Kubelet always clears reason and message in generateAPIPodStatus even when the phase is unchanged. It is reasonable that we preserve the previous values when the phase does not change, and clear it when the phase does change.

When a pod is evicted, this ensures that the eviction message and reason are propagated even in the face of subsequent updates. It also preserves the message and reason if components beyond the Kubelet choose to set that value. If reason/message are changed (due to preemption -> eviction) the most recent value is preserved.

To preserve the value we need to know the old phase, which requires a change to convertStatusToAPIStatus so that both methods have access to it.

/kind bug

Which issue(s) this PR fixes:

Fixes #103623

Does this PR introduce a user-facing change?

The reason and message fields for pod status are no longer reset unless the phase also changes.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

The Kubelet always clears reason and message in generateAPIPodStatus
even when the phase is unchanged. It is reasonable that we preserve
the previous values when the phase does not change, and clear it
when the phase does change.

When a pod is evicted, this ensurse that the eviction message and
reason are propagated even in the face of subsequent updates. It also
preserves the message and reason if components beyond the Kubelet
choose to set that value.

To preserve the value we need to know the old phase, which requires
a change to convertStatusToAPIStatus so that both methods have
access to it.
Noticed while reviewing this code path. We can assume the
temporary slice should be about the same size as it was previously.
The list of status conditions should be calculated all together,
this made review more complex. Readability only.
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. area/kubelet labels Jul 19, 2021
@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. approved Indicates a PR has been approved by an approver from all required OWNERS files. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 19, 2021
@smarterclayton
Copy link
Contributor Author

smarterclayton commented Jul 19, 2021

/assign @Random-Liu

Since we are not explicitly refreshing the pod status cache after #102344, the sequential status updates in syncTerminatedPod (once at beginning before volume detach, and once after in TerminatePod) result in clearing the message and reason during eviction. This preserves Reason/Message in the generateAPIPodStatus unless phase changes, which can occur when soft admission failures move from Running -> Pending (where we want to preserve the message even if other status updates happen).

This also makes the Kubelet more tolerant to other controllers writing reason/message, although that's not a primary goal of this change.

The third commit is a cleanup change but I can move that to the next release if it's concerning.

@smarterclayton
Copy link
Contributor Author

smarterclayton commented Jul 19, 2021

Also note the first commit ensures that soft eviction can't cause the Kubelet to attempt to change a PodSucceeded phase to a PodFailed phase

@endocrimes
Copy link
Member

endocrimes commented Jul 20, 2021

/test pull-node-kubelet-eviction

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jul 20, 2021

@endocrimes: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

  • /test pull-kubernetes-conformance-image-test
  • /test pull-kubernetes-conformance-kind-ipv6-parallel
  • /test pull-kubernetes-dependencies
  • /test pull-kubernetes-dependencies-go-canary
  • /test pull-kubernetes-e2e-ipvs-azure-dualstack
  • /test pull-kubernetes-e2e-iptables-azure-dualstack
  • /test pull-kubernetes-files-remake
  • /test pull-kubernetes-e2e-gce
  • /test pull-kubernetes-e2e-gce-no-stage
  • /test pull-kubernetes-e2e-gce-kubetest2
  • /test pull-kubernetes-e2e-gce-canary
  • /test pull-kubernetes-e2e-gce-ubuntu
  • /test pull-kubernetes-e2e-gce-ubuntu-containerd
  • /test pull-kubernetes-e2e-gce-ubuntu-containerd-canary
  • /test pull-kubernetes-e2e-gce-alpha-features
  • /test pull-kubernetes-e2e-gce-device-plugin-gpu
  • /test pull-kubernetes-integration
  • /test pull-kubernetes-integration-go-canary
  • /test pull-kubernetes-cross
  • /test check-dependency-stats
  • /test pull-kubernetes-e2e-kind
  • /test pull-kubernetes-e2e-kind-canary
  • /test pull-kubernetes-e2e-kind-ipv6
  • /test pull-kubernetes-e2e-kind-ipv6-canary
  • /test pull-kubernetes-conformance-kind-ga-only
  • /test pull-kubernetes-conformance-kind-ga-only-parallel
  • /test pull-kubernetes-e2e-kops-aws
  • /test pull-kubernetes-bazel-build-canary
  • /test pull-kubernetes-bazel-test-canary
  • /test pull-kubernetes-bazel-test-integration-canary
  • /test pull-kubernetes-local-e2e
  • /test pull-kubernetes-unit
  • /test pull-kubernetes-unit-experimental
  • /test pull-publishing-bot-validate
  • /test pull-kubernetes-e2e-aks-engine-windows-dockershim
  • /test pull-kubernetes-e2e-aks-engine-windows-containerd
  • /test pull-kubernetes-e2e-aks-engine-azure-disk-windows-dockershim
  • /test pull-kubernetes-e2e-aks-engine-azure-file-windows-dockershim
  • /test pull-kubernetes-e2e-capz-windows-dockershim
  • /test pull-kubernetes-e2e-aks-engine-gpu-windows-dockershim
  • /test pull-kubernetes-e2e-aks-engine-azure-disk-windows-containerd
  • /test pull-kubernetes-e2e-aks-engine-azure-file-windows-containerd
  • /test pull-kubernetes-e2e-capz-azure-disk
  • /test pull-kubernetes-e2e-capz-azure-disk-vmss
  • /test pull-kubernetes-e2e-capz-azure-file
  • /test pull-kubernetes-e2e-capz-azure-file-vmss
  • /test pull-kubernetes-e2e-capz-conformance
  • /test pull-kubernetes-e2e-capz-ha-control-plane
  • /test pull-kubernetes-e2e-gce-network-proxy-http-connect
  • /test pull-kubernetes-e2e-gce-network-proxy-grpc
  • /test pull-kubernetes-e2e-gci-gce-autoscaling
  • /test pull-kubernetes-e2e-kind-dual-canary
  • /test pull-kubernetes-e2e-kind-ipvs-dual-canary
  • /test pull-kubernetes-e2e-gci-gce-ingress
  • /test pull-kubernetes-e2e-ubuntu-gce-network-policies
  • /test pull-kubernetes-e2e-gci-gce-ipvs
  • /test pull-kubernetes-node-e2e
  • /test pull-kubernetes-node-e2e-podutil
  • /test pull-kubernetes-e2e-containerd-gce
  • /test pull-kubernetes-node-e2e-containerd
  • /test pull-kubernetes-node-e2e-containerd-features
  • /test pull-kubernetes-node-e2e-alpha
  • /test pull-kubernetes-node-kubelet-serial
  • /test pull-kubernetes-node-kubelet-eviction
  • /test pull-kubernetes-node-kubelet-serial-cpu-manager
  • /test pull-kubernetes-node-kubelet-serial-topology-manager
  • /test pull-kubernetes-node-kubelet-serial-hugepages
  • /test pull-kubernetes-node-crio-cgrpv2-e2e
  • /test pull-kubernetes-node-kubelet-serial-crio-cgroupv1
  • /test pull-kubernetes-node-kubelet-serial-crio-cgroupv2
  • /test pull-kubernetes-node-crio-e2e
  • /test pull-kubernetes-node-kubelet-serial-memory-manager
  • /test pull-kubernetes-node-memoryqos-cgrpv2
  • /test pull-kubernetes-node-swap-ubuntu
  • /test pull-kubernetes-node-swap-fedora
  • /test pull-kubernetes-e2e-gce-100-performance
  • /test pull-kubernetes-e2e-gce-big-performance
  • /test pull-kubernetes-e2e-gce-correctness
  • /test pull-kubernetes-e2e-gce-large-performance
  • /test pull-kubernetes-kubemark-e2e-gce-big
  • /test pull-kubernetes-kubemark-e2e-gce-scale
  • /test pull-kubernetes-e2e-gce-storage-slow
  • /test pull-kubernetes-e2e-gce-storage-snapshot
  • /test pull-kubernetes-e2e-gce-csi-serial
  • /test pull-kubernetes-e2e-gce-iscsi
  • /test pull-kubernetes-e2e-gce-iscsi-serial
  • /test pull-kubernetes-e2e-gce-storage-disruptive
  • /test pull-kubernetes-typecheck
  • /test pull-kubernetes-verify-govet-levee
  • /test pull-kubernetes-verify
  • /test pull-kubernetes-verify-go-canary
  • /test pull-kubernetes-e2e-windows-gce

Use /test all to run the following jobs:

  • pull-kubernetes-dependencies
  • pull-kubernetes-e2e-gce-ubuntu-containerd
  • pull-kubernetes-integration
  • pull-kubernetes-e2e-kind
  • pull-kubernetes-e2e-kind-ipv6
  • pull-kubernetes-conformance-kind-ga-only-parallel
  • pull-kubernetes-unit
  • pull-kubernetes-node-e2e-containerd
  • pull-kubernetes-node-crio-e2e
  • pull-kubernetes-e2e-gce-100-performance
  • pull-kubernetes-typecheck
  • pull-kubernetes-verify-govet-levee
  • pull-kubernetes-verify

In response to this:

/test pull-node-kubelet-eviction

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@endocrimes
Copy link
Member

endocrimes commented Jul 20, 2021

/test pull-kubernetes-node-kubelet-eviction

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jul 20, 2021

@smarterclayton: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Rerun command
pull-kubernetes-node-kubelet-eviction d7ee024 link /test pull-kubernetes-node-kubelet-eviction

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@endocrimes
Copy link
Member

endocrimes commented Jul 20, 2021

Looks like it fixes the related eviction failures 👍 (remaining ones need unrelated investigation)

@smarterclayton
Copy link
Contributor Author

smarterclayton commented Jul 20, 2021

Looking at some of the tests, my fixes may have corrected previous race bugs that might not have been visible:

�[1mSTEP�[0m: making sure we can start a new pod after the test
Jul 20 13:01:59.020: INFO: The status of Pod test-admit-pod is Pending, waiting for it to be Running (with Ready = true)
Jul 20 13:02:01.023: INFO: The status of Pod test-admit-pod is Failed which is unexpected

Let me try to reproduce locally here to eliminate my changes as the cause (or conversely, identify whether this is now fixed).

@Random-Liu
Copy link
Member

Random-Liu commented Jul 20, 2021

The fix makes sense to me. :)

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 20, 2021
@ehashman
Copy link
Member

ehashman commented Jul 20, 2021

/triage accepted
/priority important-soon
/assign

@k8s-ci-robot k8s-ci-robot added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Jul 20, 2021
@k8s-ci-robot k8s-ci-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 20, 2021
@ehashman ehashman added this to Needs Approver in SIG Node PR Triage Jul 20, 2021
Copy link
Member

@ehashman ehashman left a comment

[sig-node] InodeEviction [Slow] [Serial] [Disruptive][NodeFeature:Eviction] when we run containers that should cause DiskPressure should eventually evict all of the correct pods did not fail.

/lgtm
/milestone v1.22

oldPodStatus, found := kl.statusManager.GetPodStatus(pod.UID)
if !found {
oldPodStatus = pod.Status
}
Copy link
Member

@ehashman ehashman Jul 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Lifting this up one call makes a ton of sense to me.

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jul 20, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ehashman, Random-Liu, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added this to the v1.22 milestone Jul 20, 2021
@ehashman
Copy link
Member

ehashman commented Jul 20, 2021

/skip

@k8s-ci-robot k8s-ci-robot merged commit 9f47110 into kubernetes:master Jul 20, 2021
15 checks passed
SIG Node PR Triage automation moved this from Needs Approver to Done Jul 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/kubelet cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/node Categorizes an issue or PR as relevant to SIG Node. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
5 participants