Rook Operator MSG: "timeout out deleting the rbd network fence" every 2 seconds

[reefland]

Rook-Ceph operator is logging this message ~2 seconds:

2024-05-23 15:15:26.281268 I | ceph-cluster-controller: waiting for network fence CR "k3s05-rbd-rook-ceph" status to get result "unfencing operation successful"
2024-05-23 15:15:28.281561 I | ceph-cluster-controller: waiting for network fence CR "k3s05-rbd-rook-ceph" status to get result "unfencing operation successful"
2024-05-23 15:15:30.281660 I | ceph-cluster-controller: waiting for network fence CR "k3s05-rbd-rook-ceph" status to get result "unfencing operation successful"
2024-05-23 15:15:32.281215 I | ceph-cluster-controller: waiting for network fence CR "k3s05-rbd-rook-ceph" status to get result "unfencing operation successful"
2024-05-23 15:15:32.281282 E | ceph-cluster-controller: failed to handle node failure. failed to delete rbd network fence for node "k3s05".: timeout out deleting the rbd network fence CR k3s05-rbd-rook-ceph: context deadline exceeded
2024-05-23 15:19:39.568100 I | ceph-cluster-controller: node k3s05 does not have taint node.kubernetes.io/out-of-service, unfencing networkFence CR
2024-05-23 15:19:39.581378 I | ceph-cluster-controller: waiting for network fence CR "k3s05-rbd-rook-ceph" status to get result "unfencing operation successful"
2024-05-23 15:19:41.581812 I | ceph-cluster-controller: waiting for network fence CR "k3s05-rbd-rook-ceph" status to get result "unfencing operation successful"

1. This node k3s05 is up and running
2. This nodes OSDs are all UP and IN
3. Cluster is Healthy
4. The node is tainted to be storage node, perhaps a toleration is missing to allow the cleanup?

$ ceph -v
ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)

$ ceph health detail
HEALTH_OK

$ k get nodes

NAME      STATUS   ROLES                       AGE     VERSION
k3s01     Ready    control-plane,etcd,master   2y53d   v1.29.4+k3s1
k3s02     Ready    control-plane,etcd,master   2y27d   v1.29.4+k3s1
k3s03     Ready    control-plane,etcd,master   2y34d   v1.29.4+k3s1
k3s04     Ready    worker                      530d    v1.29.4+k3s1
k3s05     Ready    worker                      530d    v1.29.4+k3s1
k3s06     Ready    worker                      457d    v1.29.4+k3s1

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

NAME      TAINTS
k3s01     <none>
k3s02     <none>
k3s03     <none>
k3s04     <none>
k3s05     [map[effect:NoSchedule key:storage-node value:true]]
k3s06     <none>

$ kubectl get networkfences.csiaddons.openshift.io

NAME                  DRIVER                       CIDRS                   FENCESTATE   AGE   RESULT
k3s05-rbd-rook-ceph   rook-ceph.rbd.csi.ceph.com   ["192.168.10.219/32"]   Unfenced     55d   

$ kubectl get volumeattachments | grep -c k3s05
0

I could probably delete this manually if needed, but I'm trying to determine if something is preventing an automated cleanup.

[subhamkrai]

@reefland the logging was improved in new version of rook, what version of rook you are using?
also could logs the output of controller pod in namespace -n csi-addons-system

[reefland]

rook-version=v1.14.4
ceph-version=18.2.2-0

@subhamkrai - thanks for the reply. The csi-addons-controller was the hint I needed, not sure why but the controller was no longer deployed. I put it back and things cleaned up...

2024-05-27T15:01:49.251Z INFO UnFenceClusterNetwork Request Succeeded {"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"k3s05-rbd-rook-ceph"}, "namespace": "", "name": "k3s05-rbd-rook-ceph", "reconcileID": "e0e23e26-d94d-48a6-9d2a-830e0832afe5", "DriverName": "rook-ceph.rbd.csi.ceph.com", "CIDRs": ["192.168.10.219/32"]}
2024-05-27T15:01:49.535Z INFO removing finalizer from NetworkFence object {"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"k3s05-rbd-rook-ceph"}, "namespace": "", "name": "k3s05-rbd-rook-ceph", "reconcileID": "dcfcd12f-5d3c-42bd-bfeb-4f82df3faafc", "DriverName": "rook-ceph.rbd.csi.ceph.com", "CIDRs": ["192.168.10.219/32"], "Finalizer": "csiaddons.openshift.io/network-fence"}
2024-05-27T15:01:49.549Z INFO NetworkFence object is terminated, skipping reconciliation {"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"k3s05-rbd-rook-ceph"}, "namespace": "", "name": "k3s05-rbd-rook-ceph", "reconcileID": "dcfcd12f-5d3c-42bd-bfeb-4f82df3faafc", "DriverName": "rook-ceph.rbd.csi.ceph.com", "CIDRs": ["192.168.10.219/32"]}
2024-05-27T15:01:49.550Z INFO NetworkFence resource not found or deleted {"controller": "networkfence", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "NetworkFence", "NetworkFence": {"name":"k3s05-rbd-rook-ceph"}, "namespace": "", "name": "k3s05-rbd-rook-ceph", "reconcileID": "66a5073e-6c8c-4a27-828f-0c92cff06b41"}

$ kubectl get networkfences.csiaddons.openshift.io                                                                                                                                  1 ↵
No resources found