Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MountVolume.SetUp failed for volume pvc mount failed: exit status 32 #77916

Closed
istikamet opened this issue May 15, 2019 · 2 comments

Comments

Projects
None yet
3 participants
@istikamet
Copy link

commented May 15, 2019

/kind bug
/kind feature

We are trying to configure vsphere cloud provider for kubernetes cluster, according to below support link for dynamic provisioning from vmware. There was no problem with kubernetres version v1.8, the issue appeared after deploying new kubernetes cluster with version 1.12.
https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html

Kubernetes version
v1.12.7

Environment
Cluster: master1,node1,node2
Virtualization: VMware ESXi, 6.5.0, 7388607 , each virtual machine node placed on different esxi host, diskuuid is true
Network: Cisco Apic, Version 3.2(4e)

Problem definition:
Storage class, pv, pvc successfull created and bounded, vmdk disk added to virtual machine successfully but node cannot mount this physical disk to virtual path of the pod, there are two different log not sure which one is right, first log mention _"cloud provider not initialized"_, if not how vmdk created and attached to virtual machine, second log mention _"mount failed because not found the ...vmdk"_ but i can browse from the vcenter mentioned vmdk is in the datastore"

[root@kubemaster ~]# kubectl get pods pvpod
NAME    READY   STATUS              RESTARTS   AGE
pvpod   0/1     ContainerCreating   0          17h

kubectl descrice pod pvpod


[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl describe pod pvpod
Name:               pvpod
Namespace:          default
Priority:           0
PriorityClassName:  
Node:               kubenode2/10.1.1.12
Start Time:         Tue, 14 May 2019 12:52:18 +0300
Labels:             
Annotations:        opflex.cisco.com/computed-endpoint-group: {"policy-space":"Kubernetes","name":"kubernetes|kube-default"}
                    opflex.cisco.com/computed-security-group: []
Status:             Pending
IP:
Containers:
  test-container:
    Container ID:
    Image:          gcr.io/google_containers/test-webserver
    Image ID:
    Port:           
    Host Port:      
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    
    Mounts:
      /test-vmdk from test-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pvf97 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  test-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  pvcsc001
    ReadOnly:   false
  default-token-pvf97:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-pvf97
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age    From                           Message
  ----     ------       ----   ----                           -------
  Normal   Scheduled    8m22s  default-scheduler              Successfully assigned default/pvpod to kubenode2
  Warning  FailedMount  8m21s  kubelet, kubenode2  MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3
Output: Running scope as unit run-87284.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist
  Warning  FailedMount  8m21s  kubelet, kubenode2  MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3
Output: Running scope as unit run-87288.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist
  Warning  FailedMount  8m20s  kubelet, kubenode2  MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3
Output: Running scope as unit run-87290.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist
  Warning  FailedMount  8m18s  kubelet, kubenode2  MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk /var/lib/kubelet/pods/f63bb16e-762d-11e9-aff9-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-eb27986d-762d-11e9-aff9-005056b068e3
Output: Running scope as unit run-87329.scope.
mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-eb27986d-762d-11e9-aff9-005056b068e3.vmdk does not exist
  Normal   SuccessfulAttachVolume  8m17s  attachdetach-controller        AttachVolume.Attach succeeded for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3"
  Warning  FailedMount             8m14s  kubelet, kubenode2  MountVolume.SetUp failed for volume "pvc-eb27986d-762d-11e9-aff9-005056b068e3" : mount failed: exit status 32
  

/var/log/messages on the node

May  8 16:05:32 kubenode1 kubelet: E0508 16:05:32.730158   53933 vsphere_volume_util.go:187] Cloud provider not initialized properly
May  8 16:05:32 kubenode1 systemd: Started Kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3.
May  8 16:05:32 kubenode1 kubelet: E0508 16:05:32.744639   53933 mount_linux.go:152] Mount failed: exit status 32
May  8 16:05:32 kubenode1 kubelet: Mounting command: systemd-run
May  8 16:05:32 kubenode1 kubelet: Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3
May  8 16:05:32 kubenode1 kubelet: Output: Running scope as unit run-57611.scope.
May  8 16:05:32 kubenode1 kubelet: mount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk does not exist
May  8 16:05:32 kubenode1 kubelet: E0508 16:05:32.744865   53933 nestedpendingoperations.go:267] Operation for "\"kubernetes.io/vsphere-volume/b52b1433-718a-11e9-a3ef-005056b068e3-pvc-e3705fec-7189-11e9-a3ef-005056b068e3\" (\"b52b1433-718a-11e9-a3ef-005056b068e3\")" failed. No retries permitted until 2019-05-08 16:07:34.74479901 +0300 +03 m=+619.028655111 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"pvc-e3705fec-7189-11e9-a3ef-005056b068e3\" (UniqueName: \"kubernetes.io/vsphere-volume/b52b1433-718a-11e9-a3ef-005056b068e3-pvc-e3705fec-7189-11e9-a3ef-005056b068e3\") pod \"pvpod\" (UID: \"b52b1433-718a-11e9-a3ef-005056b068e3\") : mount failed: exit status 32\nMounting command: systemd-run\nMounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3 --scope -- mount -o bind /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk /var/lib/kubelet/pods/b52b1433-718a-11e9-a3ef-005056b068e3/volumes/kubernetes.io~vsphere-volume/pvc-e3705fec-7189-11e9-a3ef-005056b068e3\nOutput: Running scope as unit run-57611.scope.\nmount: special device /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-e3705fec-7189-11e9-a3ef-005056b068e3.vmdk does not exist\n\n"
May  8 16:05:32 kubenode1 kubelet: E0508 16:05:32.830478   53933 vsphere_volume_util.go:187] Cloud provider not initialized properly

Pv,PVC,SC yaml files


Sc-fast.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: fast
provisioner: kubernetes.io/vsphere-volume
parameters:
datastore: KUBEDATASTORE
diskformat: thin
fstype: ext3

pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvcsc001
annotations:
volume.beta.kubernetes.io/storage-class: fast
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi

pod.yaml

apiVersion: v1
kind: Pod
metadata:
name: pvpod
spec:
containers:

  • name: test-container
    image: gcr.io/google_containers/test-webserver
    volumeMounts:
    • name: test-volume
      mountPath: /test-vmdk
      volumes:
  • name: test-volume
    persistentVolumeClaim:
    claimName: pvcsc001

Output of applied yaml files


[root@kubemaster vcp]# kubectl create -f vsphere-volume-sc-fast.yaml
storageclass.storage.k8s.io/fast created
[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl describe storageclass fast
Name: fast
IsDefaultClass: No
Annotations:
Provisioner: kubernetes.io/vsphere-volume
Parameters: datastore=KUBEDATASTORE,diskformat=thin,fstype=ext3
AllowVolumeExpansion:
MountOptions:
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events:

[root@kubemaster vcp]# kubectl create -f vsphere-volume-pvcsc.yaml
persistentvolumeclaim/pvcsc001 created
[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl describe pvc pvcsc001
Name: pvcsc001
Namespace: default
StorageClass: fast
Status: Bound
Volume: pvc-4aa82312-72ec-11e9-8b74-005056b068e3
Labels:
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-class: fast
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWO
Events:
Type Reason Age From Message
Normal ProvisioningSucceeded 13s persistentvolume-controller Successfully provisioned volume pvc-4aa82312-72ec-11e9-8b74-005056b068e3 using kubernetes.io/vsphere-volume
Mounted By: <.none>

[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl describe pv pvc-4aa82312-72ec-11e9-8b74-005056b068e3
Name: pvc-4aa82312-72ec-11e9-8b74-005056b068e3
Labels:
Annotations: kubernetes.io/createdby: vsphere-volume-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume
Finalizers: [kubernetes.io/pv-protection]
StorageClass: fast
Status: Bound
Claim: default/pvcsc001
Reclaim Policy: Delete
Access Modes: RWO
Capacity: 2Gi
Node Affinity:
Message:
Source:
Type: vSphereVolume (a Persistent Disk resource in vSphere)
VolumePath: [KUBEDATASTORE] kubevols/kubernetes-dynamic-pvc-4aa82312-72ec-11e9-8b74-005056b068e3.vmdk
FSType: ext3
StoragePolicyName:
Events:


[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl create -f vsphere-volume-pvcscpod.yaml
pod/pvpod created
[root@kubemaster vcp]#
[root@kubemaster vcp]# kubectl get pod pvpod
NAME READY STATUS RESTARTS AGE
pvpod 0/1 ContainerCreating 0 49s

first progress

in the first place when i run “ vsphere-volume-pvcsc.yaml” it was not successfully creating the volume, output of the “kubectl describe pvc” message was No VM found, than i applied #65933 (comment) method to match uuids( even providerid and product serial was same), after than volume created successfully and attached into virtual machine. But Pod still ContainerCreating state.

cat /sys/class/dmi/id/product_serial | sed -e 's/^VMware-//' -e 's/-/ /' | awk '{ print toupper($1$2$3$4 "-" $5$6 "-" $7$8 "-" $9$10 "-" $11$12$13$14$15$16) }'
kubectl patch node kubemaster -p '{"spec":{"providerID":"vsphere://4230456E-D8B2-AED1-E270-740256CBD273"}}'

Second progress

We have disgussed with @shahbour at #77663 and we made sure it is not a cloud provider issue even the message "cloud provider not initialized properly" message take to the stage while container creating.

@athenabot

This comment has been minimized.

Copy link

commented May 15, 2019

/sig storage
/sig vmware

These SIGs are my best guesses for this issue. Please comment /remove-sig <name> if I am incorrect about one.

🤖 I am a bot run by vllry. 👩‍🔬

@istikamet

This comment has been minimized.

Copy link
Author

commented May 20, 2019

Problem fixed at this issue #77663 (comment)

@istikamet istikamet closed this May 20, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.