The Cinder CSI volume formated by kubernetes is not mounted

[lfdominguez]

What happened:

The Cinder CSI volume formated by kubernetes is not mounted on globalmount so when kubernetes try to mount the CSI volume inside the mount folder of the kubelet internal mount

What you expected to happen:

1- The deploy create a PVC that is bound

2- The volume is created on OpenStack Cinder (check the volume name):

3- Pod is scheduled to node k8s-3 where is the volume mounted (see prev photo):

4- The pod status show an error about failed MountVolume.SetUp in a inexistent mount file:

How to reproduce it (as minimally and precisely as possible):

Deploy a stateful set with this volumeClaimTemplates template:

volumeClaimTemplates:
    - metadata:
        name: data
        annotations:
      spec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: "20Gi"

Anything else we need to know?:

The cluster is a minik8s from canonical where the kubernetes binaries live inside /var/snap/microk8s so I changed the variables inside the CSI provider yaml files, here are the file changed (cinder-csi-nodeplugin.yaml):

# This YAML file contains driver-registrar & csi driver nodeplugin API objects,
# which are necessary to run csi nodeplugin for cinder.

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: csi-cinder-nodeplugin
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-cinder-nodeplugin
  template:
    metadata:
      labels:
        app: csi-cinder-nodeplugin
    spec:
      tolerations:
        - operator: Exists
      serviceAccount: csi-cinder-node-sa
      hostNetwork: true
      containers:
        - name: node-driver-registrar
          image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v1.3.0
          args:
            - "--csi-address=$(ADDRESS)"
            - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "rm -rf /registration/cinder.csi.openstack.org /registration/cinder.csi.openstack.org-reg.sock"]
          env:
            - name: ADDRESS
              value: /csi/csi.sock
            - name: DRIVER_REG_SOCK_PATH
              value: /var/snap/microk8s/common/var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          imagePullPolicy: "IfNotPresent"
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
        - name: liveness-probe
          image: k8s.gcr.io/sig-storage/livenessprobe:v2.1.0
          args:
            - --csi-address=/csi/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
        - name: cinder-csi-plugin
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: docker.io/k8scloudprovider/cinder-csi-plugin:latest
          args:
            - /bin/cinder-csi-plugin
            - "--nodeid=$(NODE_ID)"
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--cloud-config=$(CLOUD_CONFIG)"
          env:
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CSI_ENDPOINT
              value: unix://csi/csi.sock
            - name: CLOUD_CONFIG
              value: /etc/config/cloud.conf
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 9808
              name: healthz
              protocol: TCP
          # The probe
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: healthz
            initialDelaySeconds: 10
            timeoutSeconds: 3
            periodSeconds: 10
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: kubelet-dir
              mountPath: /var/lib/kubelet
              mountPropagation: "Bidirectional"
            - name: pods-probe-dir
              mountPath: /dev
              mountPropagation: "HostToContainer"
            - name: secret-cinderplugin
              mountPath: /etc/config
              readOnly: true
      volumes:
        - name: socket-dir
          hostPath:
            path: /var/snap/microk8s/common/var/lib/kubelet/plugins/cinder.csi.openstack.org
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/snap/microk8s/common/var/lib/kubelet/plugins_registry/
            type: Directory
        - name: kubelet-dir
          hostPath:
            path: /var/snap/microk8s/common/var/lib/kubelet
            type: Directory
        - name: pods-probe-dir
          hostPath:
            path: /dev
            type: Directory
        - name: secret-cinderplugin
          secret:
            secretName: cloud-config

Environment:

• Kubernetes version (use kubectl version):v1.20.5-34+40f5951bd9888a (microk8s)
• Cloud provider or hardware configuration: OpenStack Cinder CSI
• OS (e.g: cat /etc/os-release): Ubuntu 20.04 LTS
• Kernel (e.g. uname -a): Linux k8s-4 5.4.0-29-generic readability: error string #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: microk8s 1.20
• Network plugin and version (if this is a network-related bug):
• Others:

I0413 02:10:35.939378       1 mount_linux.go:390] Disk "/dev/disk/by-id/virtio-3434b9ec-f4bd-4d4d-8" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/disk/by-id/virtio-3434b9ec-f4bd-4d4d-8]
I0413 02:10:37.102824       1 mount_linux.go:400] Disk successfully formatted (mkfs): ext4 - /dev/disk/by-id/virtio-3434b9ec-f4bd-4d4d-8 /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-2db907ea-9715-4296-93cb-c78d3336590e/globalmount

[k8s-ci-robot]

@lfdominguez: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[lfdominguez]

/sig storage
/sig cloud-provider

[lfdominguez]

kubernetes/cloud-provider-openstack#1478 (comment)

[lfdominguez]

kubernetes/cloud-provider-openstack#1478 (comment)