Pod that failed to bind, stuck in Pending state forever

[alena1108]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

Pod got stuck in Pending state forever when failed to bind to host.

What you expected to happen:

For it to be rescheduled on another host

How to reproduce it (as minimally and precisely as possible):

Not always reproducible. But simply created an RC replicas=2 on a 3 host setup, and one of the Pods got stuck in pending state for a while. Following error messages were found in the scheduler logs:

7/20/2017 8:30:40 AME0720 15:30:40.435703       1 scheduler.go:282] Internal error binding pod: (scheduler cache ForgetPod failed: pod test-677550-rc-edit-namespace/nginx-jvn09 state was assumed on a different node)
7/20/2017 8:31:11 AMW0720 15:31:11.119885       1 cache.go:371] Pod test-677550-rc-edit-namespace/nginx-jvn09 expired

Describe pod output:

nginx-jvn09   0/1       Pending   0          2h
> kubectl describe pod nginx-jvn09 --namespace=test-677550-rc-edit-namespace
Name:           nginx-jvn09
Namespace:      test-677550-rc-edit-namespace
Node:           /
Labels:         name=nginx
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"Replicatio
nController","namespace":"test-677550-rc-edit-namespace","name":"nginx","uid":"6313354d-6d60-11e...
Status:         Pending
IP:
Controllers:    ReplicationController/nginx
Containers:
  nginx:
    Image:              nginx
    Port:               80/TCP
    Environment:        <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vz8x6 (ro)
Volumes:
  default-token-vz8x6:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-vz8x6
    Optional:   false
QoS Class:      BestEffort
Node-Selectors: <none>
Tolerations:    node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
                node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:         <none>

RC yml:

apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    name: nginx
  template:
    metadata:
      labels:
        name: nginx
    spec:
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.7.1
• Cloud provider or hardware configuration**:
• OS (e.g. from /etc/os-release): RHEL7, Docker native 1.12.6
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[k8s-merge-robot]

@alena1108
There are no sig labels on this issue. Please add a sig label by:

1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
   e.g., @kubernetes/contributor-experience-<group-suffix> to notify the contributor experience sig, OR

2. specifying the label manually: /sig <label>
   e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. You can find the group list here and label list here.
The <group-suffix> in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

[xiangpengzhao]

/sig scheduling

[jianzhangbjz]

Can you provide the details steps for this issue? And the output of kubectl describe nodes?

[dixudx]

@alena1108 The log just showed the node was notReady or unreachable.

node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s

Seems the kubelet node is not quite stable that the connections with the master always get lost.

[alena1108]

@jianglingxia @dixudx Another pod of the same RC (replicas=2) was started successfully on one of the nodes. I'd expect the failed one to at least be attempted to start again, may be on the node where its peer is running (as there are no anti affinity rules defined on it). But it was stuck in pending state forever (and there was no other reference for the failed pod besides the original log I've attached). I was able to create new RCs of the same kind on the same set of nodes successfully after the Pod scheduling failure, so it looks like some nodes were available for allocation. Perhaps there are some other logs I can look at /provide?

@jianglingxia no definite steps. We just extensive validation tests for kubernetes were a bunch of RCs/Deployments/Services/Ingress controllers get created. And starting k8s 1.7.x started observing this failure. Once we see it again, I'll fetch the node status and update the bug.

[dixudx]

@alena1108 From the log, it seemed that scheduler failed to bind your pod to the node and also failed to forget the pod, which led to unable to reschedule the pod to other nodes.

Would you please append -v==10 when you start kube-scheduler and reproduce this issue. Also the kubelet log of that erroneous node.

[alena1108]

@moelsayed ^^ could you start the schedule with the above param next time for validation test run?

[xiangpengzhao]

/cc @k82cn

[moelsayed]

I ran our tests again, and hit this issue with several pods:

$ kubectl get pods -n test-300885-testingress20
NAME                     READY     STATUS    RESTARTS   AGE
k8testrc20-one-4wqmw     1/1       Running   0          1h
k8testrc20-one-wwzmv     1/1       Running   0          1h
k8testrc20-three-hth69   1/1       Running   0          1h
**k8testrc20-three-q56k9**   0/1       Pending   0          1h
k8testrc20-two-78r3g     1/1       Running   0          1h
k8testrc20-two-ndjzm     1/1       Running   0          1h

kubectl get pods -n test-300885-testingress17
NAME                   READY     STATUS    RESTARTS   AGE
**k8testrc17-one-d2cr0**   0/1       Pending   0          1h
k8testrc17-one-xq39j   1/1       Running   0          1h

$ kubectl get pods -n test-459920-create-namespace
NAME              READY     STATUS    RESTARTS   AGE
testnginx-9xqk6   0/1       Pending   0          1h
testnginx-gngxf   1/1       Running   0          1h

There were a few more. Since none of them is assigned to a node. I included logs for all kubelet's. I also included scheduler log with --v=1:

kubelet1.log.txt
kubelet2.log.txt
kubelet3.log.txt
sched.log.txt

kubectl describe nodes:

describe_nodes.txt

[julia-stripe]

Seeing this as well.

hypothesis: ecb962e#diff-67f2b61521299ca8d8687b0933bbfb19R223 broke the error handling when ForgetPod fails. Before that commit, when ForgetPod failed it would log an error and pass the pod to the error handler (sched.config.Error(pod, err)), which is in charge of retrying scheduling the pod.

After that commit, when ForgetPod fails it skips the error handling and scheduling is never retried.

I'm doing some experiments with patching that and I think there's more to this bug than just that (for example why is ForgetPod failing in the first place? is the scheduler cache corrupted? why?), but that's what I've got so far.

[yastij]

/cc

[jianzhangbjz]

@moelsayed Can you post the description of per namespace?(kubectl describe ns xxx) Are you enabled ResourceQuota? And what's the config of scheduler component?

[alena1108]

@jianglingxia

> kubectl describe namespace test-504053-deploymentsrollback-namespace
Name:           test-504053-deploymentsrollback-namespace
Labels:         <none>
Annotations:    <none>
Status:         Active

No resource quota.

No resource limits.
>

there are no specific parameters passed on the scheduler start besides address and cloudconfig

[julia-stripe]

I posted a patch that i believe fixes this issue at #49661 -- for us it's resolved the issue so far, but I'm not very reliably able to reproduce so it's a bit hard to check. @alena1108 could you apply #49661 and see if your validation tests pass?

[alena1108]

@julia-stripe thanks!! We will be able to validate it either today or early next week

[dchen1107]

@alena1108 Are you able to validate the fix provided by @julia-stripe through #50028? Just saw this issue, and I think it is a serious regression in 1.7 release and we should patch it.

cc/ @kubernetes/kubernetes-release-managers @bsalamat @kubernetes/sig-scheduling-bugs @wojtek-t the patch manager for 1.7 patch release.

[alena1108]

@dchen1107 our QA have been busy with the current release, and didn't have a chance to validate the fix yet. Hopefully this week.

Patching sounds like a great idea. It is a regression indeed as we haven't hit this issue in the previous versions of k8s with the same set of validation tests.

[davidopp]

Thanks everyone who helped investigate this!

[alena1108]

@julia-stripe @dchen1107 just ran validation test against the patched branch - no pods stuck in Pending any more, so the fix did the job! thank you @julia-stripe

[guillelb]

@julia-stripe, I have this problem with k8s v1.7.0 .
How can I fix it, upgrading to v1.7.X?

Thank you!