[BUG] PVC expansion fails if the operator crashes in the middle of reconciliation

[srteam2020]

Describe the bug

We find that if the operator crashes at some point in the middle of performing PVC expansion, the actual PVC size will never meet user's spec even after the operator restarts.

Specifically, The rabbitmq operator performs the following steps to expand the capacity of statefulset's PVC,

1. get cluster statefulset, and if cluster statefulset exist
   1. compare capacity in CRD Spec with capacity in statefulset's volumeClaimTemplates to determine whether to resize PVC

   currentCapacity, err := persistenceStorageCapacity(current.Spec.VolumeClaimTemplates)
   ...
   desiredCapacity, err := persistenceStorageCapacity(desired.Spec.VolumeClaimTemplates)
   ...
   cmp := currentCapacity.Cmp(desiredCapacity)
   

   2. delete cluster statefulset
   3. update the actual capacity of PVC

   if err := r.deleteSts(ctx, rmq); err != nil {
   	return err
   }
   
   if err := r.updatePVC(ctx, rmq, *current.Spec.Replicas, desiredCapacity); err != nil {
   	return err
   }
   

2. create/update cluster statefulset (update PVC capacity in statefulset's volumeClaimTemplates)

We ran a workload that changes PVC capacity from 10Gi to 15Gi, and observed that PVC was never resized to 15Gi if the operator crashed at some point between deleting cluster statefulset (ii) and updating PVC capacity (iii).

In that case, the cluster statefulset was actually deleted ((ii) is executed before crash) while the actual PVC capacity is not successfully resized ((iii) is not executed before crash). After the operator restarts from the crash, it will try to get the cluster statefulset. Since the statefulset doesn't exist, the operator will skip (i),(ii),(iii), and directly create the cluster statefulset with the same PVC capacity in its volumeClaimTemplates as in the CRD spec. Since the two values become the same, the PVC expansion logic will never be triggered until the user manually change the PVC capacity in the CRD spec again.

To Reproduce

As mentioned above, resize the PVC from 10Gi to 15Gi and make the operator crash between (ii) and (iii). After restart, the PVC expansion will not succeed eventually.

Expected behavior
The operator should recover correctly from the crash and continue the PVC expansion.

Version and environment information

• RabbitMQ: 3.8.12
• RabbitMQ Cluster Operator: 4f13b9a (main branch)
• Kubernetes: 1.18.9

Additional context

We are willing to send a PR to help fix this issue. A simple fix could be to compare capacity in CRD Spec the actual PVC capacity (instead of the capacity in statefulset's volumeClaimTemplates), so that even after the untimely crash the PVC expansion can still work.

[coro]

@srteam2020 This should now be fixed, thanks again for raising the Issue!