PVC attach is failing on latest Clearlinux

[mcastelino]

  Type     Reason       Age                 From             Message
  ----     ------       ----                ----             -------
  Warning  FailedMount  74s (x34 over 75m)  kubelet, clr-03  Unable to mount volumes for pod "grafana-55dcb5484d-dq5lp_monitoring(faa0fcc4-d402-4ac1-a5fa-f8b4edcbbe68)": timeout expired waiting for volumes to attach or mount for pod "monitoring"/"grafana-55dcb5484d-dq5lp". list of unmounted volumes=[grafana-storage]. list of unattached volumes=[grafana-storage grafana-datasources grafana-dashboards grafana-dashboard-k8s-cluster-rsrc-use grafana-dashboard-k8s-node-rsrc-use grafana-dashboard-k8s-resources-cluster grafana-dashboard-k8s-resources-namespace grafana-dashboard-k8s-resources-pod grafana-dashboard-nodes grafana-dashboard-pods grafana-dashboard-statefulset grafana-token-6bv4n]

clear@clr-01 ~/clr-k8s-examples $ kubectl describe pvc -n monitoring    grafana-storage-pvc
Name:          grafana-storage-pvc
Namespace:     monitoring
StorageClass:  rook-ceph-block
Status:        Bound
Volume:        pvc-2f14f1c5-cf4f-4567-bfdb-8f1d68f5e1fc
Labels:        app=grafana
Annotations:   control-plane.alpha.kubernetes.io/leader:
                 {"holderIdentity":"888b1f2f-a35f-11e9-9ac2-b6e407c75ca1","leaseDurationSeconds":15,"acquireTime":"2019-07-10T22:10:32Z","renewTime":"2019-...
               kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"labels":{"app":"grafana"},"name":"grafana-storage-pvc","na...
               pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: ceph.rook.io/block
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    grafana-55dcb5484d-dq5lp
Events:        <none>

clear@clr-01 ~/clr-k8s-examples $ kubectl get pvc --all-namespaces -o wide
NAMESPACE     NAME                                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE   VOLUMEMODE
kube-system   elasticsearch-logging-elasticsearch-logging-0   Bound    pvc-cd198db3-0476-4f0c-9355-140e326757b8   1Gi        RWO            rook-ceph-block   85m   Filesystem
kube-system   kubernetes-dashboard-pvc                        Bound    pvc-ea813018-181e-47b7-a765-40cb6d28f1c9   1Gi        RWO            rook-ceph-block   21m   Filesystem
monitoring    alertmanager-main-db-alertmanager-main-0        Bound    pvc-8fbf3579-8d32-42a9-b31e-0355380a6b79   1Gi        RWO            rook-ceph-block   84m   Filesystem
monitoring    grafana-storage-pvc                             Bound    pvc-2f14f1c5-cf4f-4567-bfdb-8f1d68f5e1fc   1Gi        RWO            rook-ceph-block   85m   Filesystem
monitoring    prometheus-k8s-db-prometheus-k8s-0              Bound    pvc-c8a4acf3-1fd4-4011-8c39-645930e5e709   1Gi        RWO            rook-ceph-block   84m   Filesystem

clear@clr-03 ~ $ lsmod | grep rbd
rbd                    86016  0
libceph               311296  1 rbd

[mcastelino]

/cc @ganeshmaharaj @jascott1

[jascott1]

@mcastelino This should be fixed with new "wait on Rook OSD" and "wait on PVC" that came in w the Kustomize PR:
https://github.com/clearlinux/cloud-native-setup/blob/master/clr-k8s-examples/create_stack.sh#L121