Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubelet: retry pod sandbox creation when containers were never created #79451

Merged
merged 1 commit into from Jun 27, 2019

Conversation

@yujuhong
Copy link
Member

commented Jun 27, 2019

If kubelet never gets past sandbox creation (i.e., never attempted to
create containers for a pod), it should retry the sandbox creation on
failure, regardless of the restart policy of the pod.

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change

/kind bug

/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #79398

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Fix a bug where kubelet would not retry pod sandbox creation when the restart policy of the pod is Never
kubelet: retry pod sandbox creation when containers were never created
If kubelet never gets past sandbox creation (i.e., never attempted to
create containers for a pod), it should retry the sandbox creation on
failure, regardless of the restart policy of the pod.
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: yujuhong

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@yujuhong

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

The regression was introduced in 1.13. We need to patch 1.15, 1.14, and 1.13.

@yujuhong

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

The original change was introduced in #68980 /cc @derekwaynecarr

@@ -472,11 +472,15 @@ func (m *kubeGenericRuntimeManager) computePodActions(pod *v1.Pod, podStatus *ku
// If we need to (re-)create the pod sandbox, everything will need to be
// killed and recreated, and init containers should be purged.
if createPodSandbox {
if !shouldRestartOnFailure(pod) && attempt != 0 {
if !shouldRestartOnFailure(pod) && attempt != 0 && len(podStatus.ContainerStatuses) != 0 {

This comment has been minimized.

Copy link
@hex108

hex108 Jun 27, 2019

Member

Should we also add check len(podStatus.InitContainerStatuses) != 0?

This comment has been minimized.

Copy link
@yujuhong

yujuhong Jun 27, 2019

Author Member

This is a kubelet's internal type; there's no initcontianerstatuses.

This comment has been minimized.

Copy link
@yujuhong

yujuhong Jun 27, 2019

Author Member

The unit tests added show that this would work for init containers too.

@mattjmcnaughton
Copy link
Contributor

left a comment

/lgtm

modulo the open question about init containers this looks good to me! Thanks for the quick fix and for adding a test case.

@yujuhong

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

/retest

@yujuhong

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

/cc @dashpole

@k8s-ci-robot k8s-ci-robot requested a review from dashpole Jun 27, 2019

@dashpole

This comment has been minimized.

Copy link
Contributor

commented Jun 27, 2019

/lgtm
Thanks for the fix!

@yujuhong

This comment has been minimized.

Copy link
Member Author

commented Jun 27, 2019

/retest

Don't think this change would affect pull-kubernetes-kubemark-e2e-gce-big

@k8s-ci-robot k8s-ci-robot merged commit b51f621 into kubernetes:master Jun 27, 2019

23 checks passed

cla/linuxfoundation yujuhong authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-conformance-image-test Skipped.
pull-kubernetes-cross Skipped.
pull-kubernetes-dependencies Job succeeded.
Details
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-csi-serial Skipped.
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-gce-iscsi Skipped.
pull-kubernetes-e2e-gce-iscsi-serial Skipped.
pull-kubernetes-e2e-gce-storage-slow Skipped.
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-node-e2e-containerd Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details

k8s-ci-robot added a commit that referenced this pull request Jun 28, 2019

Merge pull request #79483 from yujuhong/automated-cherry-pick-of-#794…
…51-upstream-release-1.14

Automated cherry pick of #79451: kubelet: retry pod sandbox creation when containers were

k8s-ci-robot added a commit that referenced this pull request Jun 28, 2019

Merge pull request #79482 from yujuhong/automated-cherry-pick-of-#794…
…51-upstream-release-1.15

Automated cherry pick of #79451: kubelet: retry pod sandbox creation when containers were

k8s-ci-robot added a commit that referenced this pull request Jun 29, 2019

Merge pull request #79484 from yujuhong/automated-cherry-pick-of-#794…
…51-upstream-release-1.13

Automated cherry pick of #79451: kubelet: retry pod sandbox creation when containers were
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.