Kubelet rejects pod scheduled based on newly added node labels which have not been observed by the kubelet yet #93338

Joseph-Goergen · 2020-07-22T15:52:36Z

What happened:

Whenever a node is added or updated, there is a small window where pods are scheduled to that node, before any beta labels are applied to it. This can cause issues with pods that are queued up to be scheduled and that have a NodeAffinity (in our case) to the now deprecated beta.kubernetes.io/os label.

What you expected to happen:

The proper labels to be applied to workers before the scheduling of pods on that node.

How to reproduce it (as minimally and precisely as possible):

(Not 100 percent success rate)

deploy 1.19 cluster with no workers
apply a deployment with a node affinity for the beta.kubernetes.io/os label
add worker

Anything else we need to know?:

I have been told this step used to be done on the worker side, but is now done on the master side. Which could explain why this is happening. https://github.com/kubernetes/kubernetes/blob/v1.19.0-rc.2/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L1534-L1578

/sig scheduling

The text was updated successfully, but these errors were encountered:

liggitt · 2020-07-23T01:27:20Z

This can cause issues with pods that are queued up to be scheduled and that have a NodeAffinity (in our case) to the now deprecated beta.kubernetes.io/os label.

Can you describe the issue?

liggitt · 2020-07-23T01:30:38Z

The controller currently ensures the beta OS and OS labels match. What issue are you seeing occur with affinity?

rtheis · 2020-07-23T10:29:09Z

@liggitt We believe the flow is as follows:

Deploy control plane components, API server etc.
Deploy addon components (https://github.com/operator-framework/operator-lifecycle-manager in our case). These components will stay in Pending status until worker nodes are ready.
Deploy worker nodes.
Worker node X is registered with the API server (no beta.kubernetes.io/os label yet).
Scheduler schedules the OLM addon catalog pod which uses the beta.kubernetes.io/os label. This pod ends up in NodeAffinity status.
Control plan adds the beta.kubernetes.io/os label to worker node X.
Deleting the OLM addon catalog pod triggers a new pod to be created which is scheduled properly.

There is a timing window here since we don't always see the problem.

liggitt · 2020-07-23T14:44:05Z

2. Deploy addon components (https://github.com/operator-framework/operator-lifecycle-manager in our case). These components will stay in Pending status until worker nodes are ready.

Can you provide the manifests that are deployed for reference?

5. Scheduler schedules the OLM addon catalog pod which uses the beta.kubernetes.io/os label.

Can you provide the pod as fetched with kubectl get pod/$name -o yaml for reference?

5. This pod ends up in NodeAffinity status.

Are you using requiredDuringSchedulingIgnoredDuringExecution or preferredDuringSchedulingIgnoredDuringExecution affinity?

rtheis · 2020-07-24T12:26:23Z

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/psp: e2e-test-privileged-psp
  creationTimestamp: "2020-07-22T18:48:03Z"
  generateName: addon-catalog-source-
  labels:
    olm.catalogSource: addon-catalog-source
    olm.configMapResourceVersion: "10075545"
  name: addon-catalog-source-dtjxc
  namespace: ibm-system
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: CatalogSource
    name: addon-catalog-source
    uid: 1e8f8344-6f25-4dc1-b2dc-108762d9f3a7
  resourceVersion: "11390387"
  selfLink: /api/v1/namespaces/ibm-system/pods/addon-catalog-source-dtjxc
  uid: d4e741c2-fdde-49d5-b45e-e986492e9284
spec:
  containers:
  - command:
    - configmap-server
    - -c
    - addon-catalog-manifests
    - -n
    - ibm-system
    image: registry.ng.bluemix.net/armada-master/configmap-operator-registry:v1.6.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      exec:
        command:
        - grpc_health_probe
        - -addr=localhost:50051
      failureThreshold: 3
      initialDelaySeconds: 2
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    name: configmap-registry-server
    ports:
    - containerPort: 50051
      name: grpc
      protocol: TCP
    readinessProbe:
      exec:
        command:
        - grpc_health_probe
        - -addr=localhost:50051
      failureThreshold: 3
      initialDelaySeconds: 1
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 10m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: addon-catalog-source-configmap-server-token-mdcdc
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: 10.177.223.161
  nodeSelector:
    beta.kubernetes.io/os: linux
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: addon-catalog-source-configmap-server
  serviceAccountName: addon-catalog-source-configmap-server
  terminationGracePeriodSeconds: 30
  tolerations:
  - operator: Exists
  volumes:
  - name: addon-catalog-source-configmap-server-token-mdcdc
    secret:
      defaultMode: 420
      secretName: addon-catalog-source-configmap-server-token-mdcdc

rtheis · 2020-07-24T12:29:09Z

@liggitt Please see previous comment for example pod yaml and here is the OLM PR to fix the addon-catalog-source pod: operator-framework/operator-lifecycle-manager#1562. The PR details the files containing the manifests.

liggitt · 2020-07-24T12:30:03Z

nodeSelector:
    beta.kubernetes.io/os: linux

That should make the scheduler wait until a node with the appropriate labels appears to schedule the pod

Can you provide the full pod (including the status) and scheduler events which you were seeing issues with?

rtheis · 2020-07-24T15:08:36Z

@liggitt Unfortunately, I only have the following data from the previous failure. I can try to recreate the problem again if you need more information.

ibm-system    addon-catalog-source-br548                             0/1     NodeAffinity   0          147m   <none>           10.240.134.54   <none>           <none>

Name:         addon-catalog-source-br548
Namespace:    ibm-system
Priority:     0
Node:         10.240.134.54/
Start Time:   Wed, 22 Jul 2020 03:04:08 +0000
Labels:       olm.catalogSource=addon-catalog-source
              olm.configMapResourceVersion=2908
Annotations:  kubernetes.io/psp: ibm-privileged-psp
Status:       Failed
Reason:       NodeAffinity
Message:      Pod Predicate NodeAffinity failed
IP:           
IPs:          <none>
Containers:
  configmap-registry-server:
    Image:      registry.ng.bluemix.net/armada-master/configmap-operator-registry:v1.6.1
    Port:       50051/TCP
    Host Port:  0/TCP
    Command:
      configmap-server
      -c
      addon-catalog-manifests
      -n
      ibm-system
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:        10m
      memory:     50Mi
    Liveness:     exec [grpc_health_probe -addr=localhost:50051] delay=2s timeout=1s period=10s #success=1 #failure=3
    Readiness:    exec [grpc_health_probe -addr=localhost:50051] delay=1s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from addon-catalog-source-configmap-server-token-f5gsm (ro)
Volumes:
  addon-catalog-source-configmap-server-token-f5gsm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  addon-catalog-source-configmap-server-token-f5gsm
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     
Events:          <none>

kubectl get events --all-namespaces
No resources found

liggitt · 2020-07-24T15:17:27Z

@kubernetes/sig-scheduling-misc NodeAffinity failure because no nodes currently match the pod's selector is not a terminal state, right (e.g. if a node becomes available that matches the selector, the pod will be scheduled)?

ahg-g · 2020-07-24T16:37:57Z

@kubernetes/sig-scheduling-misc NodeAffinity failure because no nodes currently match the pod's selector is not a terminal state, right (e.g. if a node becomes available that matches the selector, the pod will be scheduled)?

yes, the pod should be unschedulable until a node that matches the selector shows up.

ahg-g · 2020-07-24T16:40:03Z

Relevant code: https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/framework/plugins/helper/node_affinity.go#L30-L35

invoked from the filter plugin here: https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/framework/plugins/nodeaffinity/node_affinity.go#L58-L60

ahg-g · 2020-07-24T16:42:56Z

To clarify, is the issue that a pod with nodeSelector set is getting scheduled on a node that doesn't yet have the corresponding labels?

rtheis · 2020-07-24T17:15:33Z

@ahg-g No, a pod with nodeSelector set is never getting scheduled on a node when initial scheduling attempt does not find a corresponding node label. Even when the node label is added later, the pod is stuck in NodeAffinity.

rtheis · 2020-07-24T17:16:41Z

This issue is very similar to #92067.

ahg-g · 2020-07-24T17:19:32Z

From #93338 (comment), it seems the pod got scheduled, the nodeName is set: nodeName: 10.177.223.161

ahg-g · 2020-07-24T17:24:01Z

ibm-system    addon-catalog-source-br548                             0/1     NodeAffinity   0          147m   <none>           10.240.134.54   <none>           <none>

where is this line coming from, a log? an event?

rtheis · 2020-07-24T17:46:03Z

@ahg-g #93338 (comment) is showing a successful pod. Sorry for the confusion. And I've added more context for that pod line.

+ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                                   READY   STATUS         RESTARTS   AGE    IP               NODE            NOMINATED NODE   READINESS GATES
default       test-k8s-e2e-pvg-master-verification                   1/1     Running        0          123m   172.30.3.45      10.240.134.60   <none>           <none>
ibm-system    addon-catalog-source-br548                             0/1     NodeAffinity   0          147m   <none>           10.240.134.54   <none>           <none>

ahg-g · 2020-07-24T18:00:36Z

Not sure about the source of the status field, but this is certainly not a scheduling issue.

liggitt · 2020-07-24T18:02:01Z

this is certainly not a scheduling issue.

a pod that does not have a node it can be scheduled to should remain pending, and then be scheduled successfully when such a node appears, right?

ahg-g · 2020-07-24T18:06:14Z

this is certainly not a scheduling issue.

a pod that does not have a node it can be scheduled to should remain pending, and then be scheduled successfully when such a node appears, right?

yes, and that is what is happening here, the pod didn't get scheduled initially and then got scheduled (I presume when the label got applied).

#93338 (comment) explains that the issue is that the pod didn't get scheduled after the label was applied, and I am saying that it did get scheduled because the nodeName is set.

rtheis · 2020-07-24T18:12:44Z

@ahg-g So are you saying the pod is scheduled and may be running even though status is NodeAffinity? Or maybe kubelet is confused and doesn't start the pod because it doesn't think the pod should have been scheduled to the node due to missing node label (according to kubelet)? Or maybe the API server didn't update status properly?

ahg-g · 2020-07-24T18:16:35Z

From the scheduler perspective, the pod is scheduled because the nodeName is set. The status isn't something the scheduler manages, the scheduler only adds a pod condition when it fails to schedule the pod.

ahg-g · 2020-07-24T18:19:14Z

oh, I get now where the issue is, the scheduler sees that the label is applied but kubelet doesn't, and so kubelet is not admitting the pod after the scheduler scheduled the pod.

ahg-g · 2020-07-24T18:20:56Z

/sig-node

liggitt · 2020-07-24T18:36:35Z

/sig node

@kubernetes/sig-node-bugs

ahg-g · 2020-08-27T15:33:21Z

/remove-sig scheduling

For Deployment and DaemonSet if we rely on some label not created by kubelet directly like `kubernetes.io/os` the Pod can be scheduled on a Node running a Kubelet that don't know yet the label in NodeSelector, so Pod get stuck in `NodeAffinity` and never removed. See: kubernetes/kubernetes#93338 kubernetes/kubernetes#92067 Let's rely back on `beta.kubernetes.io/os` label for the moment. NOTE: This label get deprecated in 1.19

fejta-bot · 2021-02-19T04:18:21Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

rtheis · 2021-02-19T11:22:09Z

/remove-lifecycle stale

lynic · 2021-02-25T08:31:56Z

I've met same issue with v1.19.3, reboot node normally the pod will enter NodeAffinity state.

liggitt · 2021-02-25T14:13:04Z

I've met same issue with v1.19.3, reboot node normally the pod will enter NodeAffinity state.

The fix for this issue was released to v1.19.8+ in #97996

liggitt · 2021-02-25T14:13:21Z

/close

fixed in #94087

k8s-ci-robot · 2021-02-25T14:13:30Z

@liggitt: Closing this issue.

In response to this:

/close

fixed in #94087

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Joseph-Goergen added the kind/bug Categorizes issue or PR as related to a bug. label Jul 22, 2020

k8s-ci-robot added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label Jul 22, 2020

liggitt added kind/support Categorizes issue or PR as a support question. and removed kind/bug Categorizes issue or PR as related to a bug. labels Jul 23, 2020

k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Jul 24, 2020

k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 24, 2020

liggitt changed the title ~~Node schedules pods before registration events~~ Kubelet rejects pod scheduled based on newly added node labels which have not been observed yet Jul 24, 2020

liggitt changed the title ~~Kubelet rejects pod scheduled based on newly added node labels which have not been observed yet~~ Kubelet rejects pod scheduled based on newly added node labels which have not been observed by the kubelet yet Jul 24, 2020

tnozicka mentioned this issue Aug 18, 2020

add backoff for statefulset pod to avoid fighting with kubelet when n… #92966

Closed

sjenning mentioned this issue Aug 18, 2020

kubelet waits for node lister to sync at least once #94087

Merged

k8s-ci-robot removed the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label Aug 27, 2020

dweomer mentioned this issue Sep 26, 2020

helm install job pods fails with NodeAffinity before being scheduled rancher/rke2#127

Closed

irozzo-1A mentioned this issue Nov 23, 2020

Flatcar update operator pods end-up in NodeAffinity status kubermatic/kubermatic#6185

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 19, 2021

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 19, 2021

k8s-ci-robot closed this as completed Feb 25, 2021

qiutongs mentioned this issue Mar 9, 2021

Auto-delete failed Pods #99986

Closed

alexandre-allard mentioned this issue Mar 16, 2021

Ensure bootstrap Node object is created before initial kubelet registration scality/metalk8s#2974

Closed

imharshita mentioned this issue Apr 12, 2021

Automated cherry pick of #94087: node sync at least once #97996

Merged

Kubelet rejects pod scheduled based on newly added node labels which have not been observed by the kubelet yet #93338

Kubelet rejects pod scheduled based on newly added node labels which have not been observed by the kubelet yet #93338

Joseph-Goergen commented Jul 22, 2020 •

edited

liggitt commented Jul 23, 2020

liggitt commented Jul 23, 2020

rtheis commented Jul 23, 2020

liggitt commented Jul 23, 2020 •

edited

rtheis commented Jul 24, 2020

rtheis commented Jul 24, 2020

liggitt commented Jul 24, 2020

rtheis commented Jul 24, 2020

liggitt commented Jul 24, 2020

ahg-g commented Jul 24, 2020

ahg-g commented Jul 24, 2020 •

edited

ahg-g commented Jul 24, 2020

rtheis commented Jul 24, 2020

rtheis commented Jul 24, 2020

ahg-g commented Jul 24, 2020

ahg-g commented Jul 24, 2020

rtheis commented Jul 24, 2020

ahg-g commented Jul 24, 2020

liggitt commented Jul 24, 2020

ahg-g commented Jul 24, 2020

rtheis commented Jul 24, 2020

ahg-g commented Jul 24, 2020

ahg-g commented Jul 24, 2020 •

edited

ahg-g commented Jul 24, 2020

liggitt commented Jul 24, 2020

ahg-g commented Aug 27, 2020

fejta-bot commented Feb 19, 2021

rtheis commented Feb 19, 2021

lynic commented Feb 25, 2021

liggitt commented Feb 25, 2021

liggitt commented Feb 25, 2021

k8s-ci-robot commented Feb 25, 2021

Kubelet rejects pod scheduled based on newly added node labels which have not been observed by the kubelet yet #93338

Kubelet rejects pod scheduled based on newly added node labels which have not been observed by the kubelet yet #93338

Comments

Joseph-Goergen commented Jul 22, 2020 • edited

liggitt commented Jul 23, 2020

liggitt commented Jul 23, 2020

rtheis commented Jul 23, 2020

liggitt commented Jul 23, 2020 • edited

rtheis commented Jul 24, 2020

rtheis commented Jul 24, 2020

liggitt commented Jul 24, 2020

rtheis commented Jul 24, 2020

liggitt commented Jul 24, 2020

ahg-g commented Jul 24, 2020

ahg-g commented Jul 24, 2020 • edited

ahg-g commented Jul 24, 2020

rtheis commented Jul 24, 2020

rtheis commented Jul 24, 2020

ahg-g commented Jul 24, 2020

ahg-g commented Jul 24, 2020

rtheis commented Jul 24, 2020

ahg-g commented Jul 24, 2020

liggitt commented Jul 24, 2020

ahg-g commented Jul 24, 2020

rtheis commented Jul 24, 2020

ahg-g commented Jul 24, 2020

ahg-g commented Jul 24, 2020 • edited

ahg-g commented Jul 24, 2020

liggitt commented Jul 24, 2020

ahg-g commented Aug 27, 2020

fejta-bot commented Feb 19, 2021

rtheis commented Feb 19, 2021

lynic commented Feb 25, 2021

liggitt commented Feb 25, 2021

liggitt commented Feb 25, 2021

k8s-ci-robot commented Feb 25, 2021

Joseph-Goergen commented Jul 22, 2020 •

edited

liggitt commented Jul 23, 2020 •

edited

ahg-g commented Jul 24, 2020 •

edited

ahg-g commented Jul 24, 2020 •

edited