Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubelet doesn't restart static pods after eviction #76598

Open
namreg opened this Issue Apr 15, 2019 · 6 comments

Comments

Projects
None yet
5 participants
@namreg
Copy link
Contributor

namreg commented Apr 15, 2019

What happened:
Kubelet does not restart static pods after eviction.

What you expected to happen:
Kubelete restarts static pods when eviction condition has not already met.

How to reproduce it (as minimally and precisely as possible):
Create a static pod in the pod manifests folder with the following spec:

---
apiVersion: v1
kind: Pod
metadata:
  name: test-static-pod
  namespace: kube-system
  labels:
    app: test-static-pod
spec:
  restartPolicy: Always
  containers:
    - name: pause
      image: "nginx"
      imagePullPolicy: IfNotPresent

Then, for example, emulate DiskPressure node condition:

fallocate -l 100G /test

Make sure test-static-pod was evicted:

kubectl get pods test-static

After that, address DiskPressure condition by removing test file:

rm -f /test

Make sure DiskPressure condition is disappeared from node:

kubectl describe node nodename -o yaml | grep -A5 Conditions

Check test-static-pod status:

kubectl get pods test-static-pod

The pod still has status Evicted

Anything else we need to know?:
Similar issue #71134

Environment:

@namreg

This comment has been minimized.

Copy link
Contributor Author

namreg commented Apr 15, 2019

@kubernetes/sig-node-bugs

@k8s-ci-robot k8s-ci-robot added sig/node and removed needs-sig labels Apr 15, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Apr 15, 2019

@namreg: Reiterating the mentions to trigger a notification:
@kubernetes/sig-node-bugs

In response to this:

@kubernetes/sig-node-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mattjmcnaughton

This comment has been minimized.

Copy link
Contributor

mattjmcnaughton commented Apr 15, 2019

/assign mattjmcnaughton

I'd love to take a shot at debugging and fixing this if its open. If not, feel free to cancel my assignment and take over :)

@hprateek43

This comment has been minimized.

Copy link
Contributor

hprateek43 commented Apr 15, 2019

@mattjmcnaughton Hey Matt, I am new here. Can you give me a small head start on what probably is the issue here and what might be a good patch for this condition? Not looking to take over, but looking forward to some leads in the code where I can understand this issue further.

@mattjmcnaughton

This comment has been minimized.

Copy link
Contributor

mattjmcnaughton commented Apr 16, 2019

@FX-HAO

This comment has been minimized.

Copy link

FX-HAO commented Apr 19, 2019

I think that kubelet currently doesn't rerun evicted pods after the node goes under pressure from out of resource. I don't know whether it's the design on purpose, I am still learning the codebase :P

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.