Cant mount pvc to pod not multiple to 1Gi

[lelvisl]

Describe the bug

Events:
  Type     Reason                  Age                    From                     Message
  ----     ------                  ----                   ----                     -------
  Warning  FailedScheduling        5m55s (x7 over 5m55s)  default-scheduler        pod has unbound immediate PersistentVolumeClaims (repeated 10 times)
  Normal   Scheduled               5m55s                  default-scheduler        Successfully assigned services/coredns-etcd-cluster-tzmfjstfm7 to worker9
  Normal   SuccessfulAttachVolume  5m55s                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-a54a3182-3485-11e9-907f-0050562c0156"
  Warning  FailedMount             101s (x10 over 5m51s)  kubelet, worker9         MountVolume.SetUp failed for volume "pvc-a54a3182-3485-11e9-907f-0050562c0156" : rpc error: code = Unknown desc = exit status 1
  Warning  FailedMount             98s (x2 over 3m52s)    kubelet, worker9         Unable to mount volumes for pod "coredns-etcd-cluster-tzmfjstfm7_services(a54b046c-3485-11e9-907f-0050562c0156)": timeout expired waiting for volumes to attach or mount for pod "services"/"coredns-etcd-cluster-tzmfjstfm7". list of unmounted volumes=[etcd-data]. list of unattached volumes=[etcd-data]

Kubernetes and Ceph CSI Versions
Kubernetes: 1.13
Ceph CSI: 1.0.0

Ceph CSI Driver logs
$ k logs csi-rbdplugin-fxd79 -n ceph csi-rbdplugin | grep pvc-a54a3182-3485-11e9-907f-0050562c0156

E0219 20:34:18.277187  119095 mount_linux.go:496] format of disk "/dev/rbd2" failed: type:("ext4") target:("/var/lib/kubelet/pods/a54b046c-3485-11e9-907f-0050562c0156/volumes/kubernetes.io~csi/pvc-a54a3182-3485-11e9-907f-0050562c0156/mount") options:(["defaults"])error:(exit status 1)
I0219 20:34:50.355002  119095 utils.go:98] GRPC request: {"secrets":"***stripped***","target_path":"/var/lib/kubelet/pods/a54b046c-3485-11e9-907f-0050562c0156/volumes/kubernetes.io~csi/pvc-a54a3182-3485-11e9-907f-0050562c0156/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"adminid":"admin","csiNodePublishSecretName":"csi-rbd-secret","csiNodePublishSecretNamespace":"ceph","csiProvisionerSecretName":"csi-rbd-secret","csiProvisionerSecretNamespace":"ceph","imageFeatures":"layering","imageFormat":"2","monitors":"10.35.12.24:6789,10.35.12.25:6789,10.35.12.29:6789,10.35.12.30:6789","pool":"rbd","storage.kubernetes.io/csiProvisionerIdentity":"1550146629167-8081-","userid":"kubernetes"},"volume_id":"csi-rbd-vol-a551735b-3485-11e9-b5d8-82e9bc3633cf"}
E0219 20:34:50.377908  119095 mount_linux.go:496] format of disk "/dev/rbd2" failed: type:("ext4") target:("/var/lib/kubelet/pods/a54b046c-3485-11e9-907f-0050562c0156/volumes/kubernetes.io~csi/pvc-a54a3182-3485-11e9-907f-0050562c0156/mount") options:(["defaults"])error:(exit status 1)
I0219 20:35:54.463222  119095 utils.go:98] GRPC request: {"secrets":"***stripped***","target_path":"/var/lib/kubelet/pods/a54b046c-3485-11e9-907f-0050562c0156/volumes/kubernetes.io~csi/pvc-a54a3182-3485-11e9-907f-0050562c0156/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"adminid":"admin","csiNodePublishSecretName":"csi-rbd-secret","csiNodePublishSecretNamespace":"ceph","csiProvisionerSecretName":"csi-rbd-secret","csiProvisionerSecretNamespace":"ceph","imageFeatures":"layering","imageFormat":"2","monitors":"10.35.12.24:6789,10.35.12.25:6789,10.35.12.29:6789,10.35.12.30:6789","pool":"rbd","storage.kubernetes.io/csiProvisionerIdentity":"1550146629167-8081-","userid":"kubernetes"},"volume_id":"csi-rbd-vol-a551735b-3485-11e9-b5d8-82e9bc3633cf"}
E0219 20:35:54.488375  119095 mount_linux.go:496] format of disk "/dev/rbd2" failed: type:("ext4") target:("/var/lib/kubelet/pods/a54b046c-3485-11e9-907f-0050562c0156/volumes/kubernetes.io~csi/pvc-a54a3182-3485-11e9-907f-0050562c0156/mount") options:(["defaults"])error:(exit status 1)

To Reproduce

add etcd-operator and apply this yaml

apiVersion: "etcd.database.coreos.com/v1beta2"
kind: "EtcdCluster"
metadata:
  name: "coredns-etcd-cluster"
  annotations:
    etcd.database.coreos.com/scope: clusterwide
spec:
  size: 3
  version: "3.2.13"
  pod:
    persistentVolumeClaimSpec:
      storageClassName: csi-rbd
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Mi

[lelvisl]

I changed storage to 1Gi and it works.

As i see here

ceph-csi/pkg/rbd/controllerserver.go

Line 131 in 07d26e6

volSizeGB := int(volSizeBytes / 1024 / 1024 / 1024)

you calculate in Gb and always get rounded result (zero in case volSizeBytes lower than 1Gb) - https://play.golang.org/p/jKVGCud8Jsi

[rootfs]

the min size is very fs type dependent

[lelvisl]

yes, but we are talking about rbd, fs goes after.

[Madhu-1]

@lelvisl this is fixed now, can you verify and close this one?

[lelvisl]

one or two days for tests on our cluster

[lelvisl]

Yes, it works. Thanks!

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  volumes:
  - name: data
    persistentVolumeClaim:
     claimName: myapp
  containers:
  - name: myapp-container
    image: busybox
    command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600']
    volumeMounts:
    - name: data
      mountPath: /data/
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 300Mi
  storageClassName: csi-rbd

$ k exec -it myapp-pod sh                                                                                                                                                                                            (autobp/ceph)
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  30.9G     16.9G     12.7G  57% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                    31.3G         0     31.3G   0% /sys/fs/cgroup
/dev/rbd3               282.5M      2.0M    276.5M   1% /data
/dev/mapper/INTVG-var
                         30.9G     16.9G     12.7G  57% /dev/termination-log
/dev/mapper/INTVG-var
                         30.9G     16.9G     12.7G  57% /etc/resolv.conf
/dev/mapper/INTVG-var
                         30.9G     16.9G     12.7G  57% /etc/hostname
/dev/mapper/INTVG-var
                         30.9G     16.9G     12.7G  57% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                    31.3G     12.0K     31.3G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                    31.3G         0     31.3G   0% /proc/acpi
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/keys
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/timer_stats
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                    31.3G         0     31.3G   0% /proc/scsi
tmpfs                    31.3G         0     31.3G   0% /sys/firmware
/ #