"Toy" cluster: "error getting secret rook-csi-rbd-provisioner"

[ook]

I have a toy/test rook cluster with 2 nodes (2 mgr, 2 mon) with

  useAllNodes: false
  useAllDevices: false

And the different devices allowed declared.

The replicapool is configured to size: 1, with requireSafeReplicaSize: false set, and the storage class use that weak pool.

When requiring a PVC, I see this error in provisioner:

W0104 18:51:53.386086       1 controller.go:934] Retrying syncing claim "1062425e-a4c7-409a-99e3-3e1eec510b4f", failure 0
I0104 18:51:53.386140       1 event.go:298] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pg", Name:"data-postgresql-0", UID:"1062425e-a4c7-409a-99e3-3e1eec510b4f", APIVersion:"v1", ResourceVersion:"16875", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "rook-ceph-block-weak": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
E0104 18:51:53.386183       1 controller.go:957] error syncing claim "1062425e-a4c7-409a-99e3-3e1eec510b4f": failed to provision volume with StorageClass "rook-ceph-block-weak": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found

Indeed, the secret is missing:

rook-ceph     rook-ceph-admin-keyring             kubernetes.io/rook   1      70m
rook-ceph     rook-ceph-config                    kubernetes.io/rook   2      70m
rook-ceph     rook-ceph-mon                       kubernetes.io/rook   4      70m
rook-ceph     rook-ceph-mons-keyring              kubernetes.io/rook   1      70m
rook-ceph     sh.helm.release.v1.rook-ceph.v1     helm.sh/release.v1   1      78m

Could you help me to fix this error?
Thank you.

[travisn]

Do you see the csi-rbdplugin-provisioner pods running? The secret won't be created until the csi driver is started.

[ook]

Thank you @travisn, here the pods in rook-ceph namespace, the rbd provisionner are up and running, and the error above comes from the 9qgck rbd rbd provisioner pod.

rook-ceph     csi-cephfsplugin-6lppj                          2/2     Running     0              113m
rook-ceph     csi-cephfsplugin-provisioner-6797886cc9-2j42q   5/5     Running     1 (112m ago)   113m
rook-ceph     csi-cephfsplugin-provisioner-6797886cc9-5jvzz   5/5     Running     0              113m
rook-ceph     csi-cephfsplugin-x9pdp                          2/2     Running     0              113m
rook-ceph     csi-rbdplugin-67tl5                             2/2     Running     0              113m
rook-ceph     csi-rbdplugin-provisioner-7b9c4fb44c-8n22p      5/5     Running     0              113m
rook-ceph     csi-rbdplugin-provisioner-7b9c4fb44c-9qgck      5/5     Running     0              113m
rook-ceph     csi-rbdplugin-tpbft                             2/2     Running     0              113m
rook-ceph     rook-ceph-mon-a-5c55b78589-tt4td                2/2     Running     0              113m
rook-ceph     rook-ceph-mon-b-6df576b95b-2x5p4                2/2     Running     0              91m
rook-ceph     rook-ceph-operator-7c87dbf59d-hnpms             1/1     Running     0              121m

[travisn]

@Rakshith-R ptal

[ook]

So I understand it's less trivial than I expected. Some more details to help the resolution:
Cluster is made of Raspberry 4 and Raspberry 3, so ARM64 images (debian based, kernel 6.1 recompiled with rbd module). Kubernetes v1.28 is k3s (so containerd, not docker) installed via k3sup.

Here the cluster yaml (requiring a ceph17) : https://paste.ookook.fr/paste/LZb00qqd#cwwoSDfOwxpsSYs8v0sHM8YfWtRTMn5FPmt9tHqoHrg
And the replicapool / storageclass : https://paste.ookook.fr/paste/6i2wEXC2#Dny8Ajdx-dfd7J5MWtB9f9ciQKgj4jcDEvlC7DzhDgs

Here the operator logs: https://paste.ookook.fr/paste/jsv-Ht-M#FxYJOblN92TsfocUTJr+s7zvLhsFV6uvm5Wd8aBs6ZW (we can see the cluster regulary fails but heal each time)

Here mon.a logs: https://copy-paste.online/?code=7681
Here mon.b logs: https://copy-paste.online/?code=7812

And csi-rbdplugin-provisioner: https://paste.ookook.fr/paste/onwcXdze#YD6ttjLSNjZPDSMA1caPiv-x2+PkrpasnOIMi4qxT6c

Don't hesitate to request more information to help.

[Madhu-1]

@ook it looks like there is some problem with the ceph cluster creation, once the ceph cluster is in OK state, the ceph user and ceph secretes will get created. please check the cephcluster status before creating the PVC