core: disable mirroring on pool with "image" mode

[sp98]

For image mode mirroring, if cephBlockPool.Pool.Spec.Mirroring.Enable is set to false, then remove the peer cluster and disable mirroring on all the pool if the user has disabled mirroring on all the pool images.

if mirroring is not disabled on all the pool images, then reconcile will fail asking the users to manually disable mirroring on those images.

Some initial Testing:

When mirroring is enabled:
--------------------

runner@fv-az1017-987:~/work/rook/rook$ oc get cephblockpool replicapool -o yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  creationTimestamp: "2024-03-15T04:35:27Z"
  finalizers:
  - cephblockpool.ceph.rook.io
  generation: 3
  name: replicapool
  namespace: rook-ceph
  resourceVersion: "6053"
  uid: ee1db641-1657-47cd-a031-d08c44f492ca
spec:
  application: ""
  erasureCoded:
    codingChunks: 0
    dataChunks: 0
  failureDomain: osd
  mirroring:
    enabled: true
    mode: image
    peers:
      secretNames:
      - pool-peer-token-replicapool-config
  quotas: {}
  replicated:
    size: 1
  statusCheck:
    mirror: {}
status:
  info:
    rbdMirrorBootstrapPeerSecretName: pool-peer-token-replicapool
  mirroringInfo:
    lastChanged: "2024-03-15T04:54:23Z"
    lastChecked: "2024-03-15T04:55:23Z"
    mode: image
    peers:
    - client_name: client.rbd-mirror-peer
      direction: rx-tx
      mirror_uuid: 910ebd07-140c-4415-8cde-de715efcc35a
      site_name: b6a29137-eaa6-4a7a-a5a5-b9577d63309c
      uuid: 5221e1e2-38f2-49a6-8d08-0e47a1937655
    site_name: 357462f6-0c53-4b02-8c5a-763b1f500cff
  mirroringStatus:
    lastChecked: "2024-03-15T04:55:23Z"
    summary:
      daemon_health: OK
      health: OK
      image_health: OK
      states:
        replaying: 1
  observedGeneration: 1
  phase: Ready
  snapshotScheduleStatus: {}
  
  
  
  After disabling mirroring on each pool image and then disabling mirroring on cephCluster spec.
  ---------------------------
  
  runner@fv-az1017-987:~/work/rook/rook$ oc get cephblockpool replicapool -o yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  creationTimestamp: "2024-03-15T04:35:27Z"
  finalizers:
  - cephblockpool.ceph.rook.io
  generation: 4
  name: replicapool
  namespace: rook-ceph
  resourceVersion: "6232"
  uid: ee1db641-1657-47cd-a031-d08c44f492ca
spec:
  application: ""
  erasureCoded:
    codingChunks: 0
    dataChunks: 0
  failureDomain: osd
  quotas: {}
  replicated:
    size: 1
  statusCheck:
    mirror: {}
status:
  mirroringInfo:
    lastChanged: "2024-03-15T04:57:23Z"
  mirroringStatus: {}
  observedGeneration: 4
  phase: Ready
  snapshotScheduleStatus: {}
runner@fv-az1017-987:~/work/rook/rook$ 


-------------

runner@fv-az1017-987:~/work/rook/rook$ oc exec -it rook-ceph-tools-848bb44cb-w44zh sh 
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-4.4$ ceph status  
  cluster:
    id:     357462f6-0c53-4b02-8c5a-763b1f500cff
    health: HEALTH_OK
 
  services:
    mon:           1 daemons, quorum a (age 28m)
    mgr:           a(active, since 22m)
    mds:           1/1 daemons up, 1 hot standby
    osd:           1 osds: 1 up (since 27m), 1 in (since 28m)
    cephfs-mirror: 1 daemon active (1 hosts)
    rbd-mirror:    1 daemon active (1 hosts)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 144 pgs
    objects: 42 objects, 1.2 MiB
    usage:   50 MiB used, 6.0 GiB / 6 GiB avail
    pgs:     144 active+clean
 
  io:
    client:   853 B/s rd, 1 op/s rd, 0 op/s wr
 
sh-4.4$ rbd mirror pool status replicapool
rbd: mirroring not enabled on the pool
sh-4.4$ 

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[parth-gr]

multi cluster mirroring ci test is failing

[sp98]

multi cluster mirroring ci test is failing

Still testing it with the CI. So added a wait:

        run : |
            sleep 1h```

And I just stop the CI in sometime. That causes it to fail. Otherwise it was working fine.

[travisn]

There could be a lot of images. I wonder if we need to clean this up in a job, similar to how it's proposed for force deletion of the subvolume groups.

[sp98]

made changes to disable mirroring on only those images were it was enabled.

[sp98]

Since we still don't have the force deletion of the subvolume PR yet, I propose that disabling mirroring for images in a job (if required), can be taken up once the other PR is ready.

[Madhu-1]

LGTM

[sp98]

so we are still pondering on whether we should disable mirroring on the images in this PR

@travisn @Madhu-1 thoughts?

[Madhu-1]

so we are still pondering on whether we should disable mirroring on the images in this PR

@travisn @Madhu-1 thoughts?

IMO we should not do it internally until users want us to do it with some extra flags/annotations, this could be a separate PR's

[Madhu-1]

i forgot one case where we want to allow users to remove unwanted(dead) cluster mirroring peers.

Only a few minor questions remain, removing review to unblock after others approve

[sp98]

i forgot one case where we want to allow users to remove unwanted(dead) cluster mirroring peers.

Currently, if mirroring is disabled by the customer, then we remove all the peers on the pool (provided that the mirroring on the pool images is disabled).

We don't have the automation to only remove specific/unwanted/dead peers as of now. But that looks like a valid use case and is not very difficult to implement. We can discuss more about this and maybe take it up in a future PR.

[Madhu-1]

LGTM

[parth-gr]

if we always need to disable it, do the code after this ever hit,
Or the flow is,
Disable mirroring -> disable mirroring manually on images-> the re-start reconcile

[sp98]

For image mode mirroring, we only want to disable it only when mirroring is disable on all the images on the pool.

len(*mirroredPools.Images) > 0 would mean that there are still images in this pool with mirroring enabled. So we will return the error with this message. Reconcile will keep on failing until the users manually disable the mirroring on each image on this pool.