Rook recreating purged OSDs

[mliker]

Hi,

I have purged several OSDs from the cluster (rook-operator scaled down, deployment osd.X scaled down, then ceph osd out, ceph osd purge, deployment osd.X deleted), rebalance, then move onto next osd.

After working on this for a day or two and purging all OSD I wanted, I removed the devices that the purged OSD were using from cephcluster CRD and then scaled up the operator at which point it recreated all purged OSD deployments. These are now failing of course since they have been purged from ceph. ceph osd tree does not contain any purged OSD

I also tried using removeOSDsIfOutAndSafeToRemove: true but to no avail.

Trying to purge the OSD again using the rook-ceph kubectl plugin it complains that it can not find it in osddump

$ kubectl rook-ceph rook purge-osd 0 --force
Info: Running purge osd command
2024-03-04 10:25:17.815902 W | cephcmd: loaded admin secret from env var ROOK_CEPH_SECRET instead of from file
2024-03-04 10:25:17.815954 I | rookcmd: starting Rook v1.13.4 with arguments 'rook ceph osd remove --osd-ids=0 --force-osd-removal=true'
2024-03-04 10:25:17.815956 I | rookcmd: flag values: --force-osd-removal=true, --help=false, --log-level=INFO, --osd-ids=0, --preserve-pvc=false
2024-03-04 10:25:17.815957 I | ceph-spec: parsing mon endpoints: m=172.30.93.194:6789,g=172.30.255.30:6789,k=172.30.236.240:6789
2024-03-04 10:25:17.829171 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2024-03-04 10:25:17.829260 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2024-03-04 10:25:18.026227 I | cephosd: validating status of osd.0
2024-03-04 10:25:18.026279 C | rookcmd: failed to get osd status for osd 0: not found osd.0 in OSDDump
Error: command terminated with exit code 1

this is rook v1.13.4 and ceph 18.2.0

[subhamkrai]

@mliker do you have the matching osd count in cephcluster up?

Generally, I have seen this error when someone removed a osd but the osd-count in cephCLuster doesn't match with the osd that are up

[mliker]

Hi @subhamkrai thanks for getting back to me. I realised that I hadn't cleaned up the drives themselves so the prepare jobs were re-discovering them. Now the OSDs are no longer recreated when I scale up the rook operator