PVC is not annotated with "volume.kubernetes.io/selected-node" when .spec.nodeName is used

[ialidzhikov]

What happened:
PVC stays forever in Pending state when the Pod that uses the PVC is created with .spec.nodeName.

What you expected to happen:
I would expect to be able to schedule a Pod on a predefined Node (using .spec.nodeName) that use a PVC (kubelet running with --cloud-provider=external and CSI migration feature gates are enabled on the corresponding components).

How to reproduce it (as minimally and precisely as possible):

1. The default storage class is the following one:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: default
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

2. Apply a PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ebs-claim
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 4Gi

3. Apply a Pod with .spec.nodeName that uses this PVC

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: centos
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: ebs-claim
  nodeName: ip-10-222-0-45.eu-west-1.compute.internal

5. Ensure that the PVC does not have the volume.kubernetes.io/selected-node annotation.
   The Pod stays in ContainerCreating with events:

$ k describe po app

  Warning  FailedMount  3m40s (x300 over 93m)  kubelet, ip-10-222-0-45.eu-west-1.compute.internal  Unable to attach or mount volumes: unmounted volumes=[persistent-storage], unattached volumes=[persistent-storage default-token-65nsk]: error processing PVC default/ebs-claim-coreos: PVC is not bound

The PVC events

$ k describe pvc ebs-claim

Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
  Normal  WaitForFirstConsumer  100s (x382 over 96m)  persistentvolume-controller  waiting for first consumer to be created before binding

kube-controller-manager is waiting forever for the volume.kubernetes.io/selected-node annotation to be added on the PVC:

I0408 11:09:43.546351       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"f93a5a92-4ef8-4625-a8eb-5124ff3cd24f", APIVersion:"v1", ResourceVersion:"11588", FieldPath:""}): type: 'Normal' reason: 'WaitForFirstConsumer' waiting for first consumer to be created before binding
I0408 11:09:58.546487       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim", UID:"f93a5a92-4ef8-4625-a8eb-5124ff3cd24f", APIVersion:"v1", ResourceVersion:"11588", FieldPath:""}): type: 'Normal' reason: 'WaitForFirstConsumer' waiting for first consumer to be created before binding
I0408 11:09:58.546514       1 event.go:278] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"ebs-claim-coreos", UID:"a9d72c3c-1440-4780-ba6e-660b1a0e2679", APIVersion:"v1", ResourceVersion:"14685", FieldPath:""}): type: 'Normal' reason: 'WaitForFirstConsumer' waiting for first consumer to be created before binding

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.18.0
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

[ialidzhikov]

/sig storage
/sig scheduling
/kind regression

[liggitt]

/remove-kind regression

before adding this label, please provide information about the specific previous release where the behavior you are expecting worked differently

[ialidzhikov]

I tested the steps to reproduce with v1.17.4 + in-tree volume drivers and the result is the same. So my assumption was that this is a regression coming with CSI, but it looks like it is not the case.
Thanks for the note @liggitt .

[liggitt]

thanks for the update

/remove-sig scheduling

when specifying .spec.nodeName, scheduling is not involved

[ialidzhikov]

Ok, but isn't the kube-scheduler responsible for setting the volume.kubernetes.io/selected-node annotation?

[ucabsdp]

I've also encountered similar issues with PVCs which just started today, autoscaling pods created via HPA are staying in ContainerCreating state with issue:
Warning FailedAttachVolume 29m attachdetach-controller Multi-Attach error for volume
Warning FailedMount 33s (x13 over 27m) kubelet, gke-test-default-pool-856721bc-0slq Unable to mount volumes for pod "instance-1-webapp-deployment-6564fd95cb-4zhpx_default(3d110b32-d3bb-454c-92ad-ad92fac3e07e)": timeout expired waiting for volumes to attach or mount for pod "default"/"instance-1-webapp-deployment-6564fd95cb-4zhpx"

[ialidzhikov]

@ucabsdp , there is no error message like timeout expired waiting for volumes to attach or mount for pod in the description of this issue. You can simply open a new issue for the things you observe. Thanks!

[mikechengwei]

@ialidzhikov I also meet similar problem. I find specify nodeName will skip volume mount operation reference openebs/openebs#2915

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ialidzhikov]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[msau42]

this is expected behavior. Instead of setting nodeName directly, set nodeSelector

[msau42]

/close

[k8s-ci-robot]

@msau42: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gopida]

@ucabsdp how did you fix the 'ContainerCreating' issue? Happens when pods is created in a different node other than pvc.