Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pod with restartPolicy: Always was not restarted #85831

Closed
lojies opened this issue Dec 3, 2019 · 6 comments

Comments

@lojies
Copy link
Contributor

@lojies lojies commented Dec 3, 2019

What happened:
Pod was not restarted with error:

predicate.go:75] Failed to admit pod openpalette-broker-924cbc0a-1d08-47c3-968e-bcb4fc5a0cd1-1-zhq2d_opcs(68ff947d-12f8-11ea-af20-744aa400db46) - Update plugin resources failed due to failed to write checkpoint file "kubelet_internal_checkpoint": write /var/lib/kubelet/device-plugins/.449002141: no space left on device, which is unexpected.
predicate.go:75] Failed to admit pod iag-192.113.5.13_kube-system(4272d198e0e30f0de9f5eef7d0799ebc) - Update plugin resources failed due to failed to write checkpoint file "kubelet_internal_checkpoint": write /var/lib/kubelet/device-plugins/.525042640: no space left on device, which is unexpected.

openpalette-broker-924cbc0a-1d08-47c3-968e-bcb4fc5a0cd1-1-zhq2d is a pod with nodeAffinity.

NAME                                                              READY   STATUS                     RESTARTS   AGE
openpalette-broker-924cbc0a-1d08-47c3-968e-bcb4fc5a0cd1-1-zhq2d   0/1     UnexpectedAdmissionError   0          3d2h

iag-192.113.5.13 is a static pod.
What you expected to happen:
Pod can be restarted.
How to reproduce it (as minimally and precisely as possible):
This accoured when disk is full.Some pods can't be restarted when disk is free.
Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version):
    1.13.6

Other:
I'm not sure if this is similar with #72202

@lojies lojies added the kind/bug label Dec 3, 2019
@lojies

This comment has been minimized.

Copy link
Contributor Author

@lojies lojies commented Dec 3, 2019

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node and removed needs-sig labels Dec 3, 2019
@tedyu

This comment has been minimized.

Copy link
Contributor

@tedyu tedyu commented Dec 3, 2019

Some pods can't be restarted when disk is free

Were the pods on the node where disk was full ?

@lojies

This comment has been minimized.

Copy link
Contributor Author

@lojies lojies commented Dec 3, 2019

@tedyu, yes.

@zouyee

This comment has been minimized.

Copy link
Member

@zouyee zouyee commented Dec 3, 2019

/var/lib/kubelet/device-plugins/.525042640 no space left on device

@liggitt

This comment has been minimized.

Copy link
Member

@liggitt liggitt commented Dec 3, 2019

A restart policy means the kubelet will attempt to restart the containers (which it is doing). error conditions (like the disk being full) can prevent the restart from succeeding. Based on the error, you need to free space on the disk.

@liggitt liggitt closed this Dec 3, 2019
@lojies

This comment has been minimized.

Copy link
Contributor Author

@lojies lojies commented Dec 4, 2019

A restart policy means the kubelet will attempt to restart the containers (which it is doing). error conditions (like the disk being full) can prevent the restart from succeeding. Based on the error, you need to free space on the disk.

But when i free the disk, the pod is still not restarted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.