Object buckets gone after upgrade from v1.5.1 to v1.5.2

[maximilize]

Is this a bug report or feature request?

• Bug Report

I have a rook-ceph cluster running with cephfs, block storage and object store enabled. I set it up just recently with version v1.5.1 and used most of the time the example manifests.
After applying the v1.5.1 manifests, the s3 buckets are gone inside radosgw, but the CRD's still exists.

It seems that the rook operator got stuck in a loop of re-creating and deleting the buckets.

Expected behavior:

Applying the upgrade to v1.5.2 without any issues, especially without deleting all data.

How to reproduce it (minimal and precise):

1. Setup a rook-ceph v1.5.1 cluster with object storage enabled.
2. Create one or more buckets. I used the storage class rook-ceph-delete-bucket
3. Apply upgrades to v1.5.2:

kubectl apply -f https://raw.githubusercontent.com/rook/rook/v1.5.2/cluster/examples/kubernetes/ceph/crds.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/v1.5.2/cluster/examples/kubernetes/ceph/common.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/v1.5.2/cluster/examples/kubernetes/ceph/operator.yaml
kubectl apply -f https://raw.githubusercontent.com/rook/rook/v1.5.2/cluster/examples/kubernetes/ceph/toolbox.yaml

4. After a while the buckets are gone. Check with radosgw-admin bucket ls. The CRD's still exists

File(s) to submit:
ceph-s3-crash.log (Operator log)

Environment:

• OS (e.g. from /etc/os-release): Ubuntu 20.04
• Cloud provider or hardware configuration: Bare metal
• Kubernetes version (use kubectl version): v1.19.3
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): Rancher RKE
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): Healthy

[BlaineEXE]

Having taken a little break and come back to this, I think this is a result of a bug that is fixed in v1.5.2. For creating new object bucket claims using Rook v1.5.2, the behavior should work properly. But when upgrading, the bug existing before v1.5.2 is exacerbated/exposed.

I think the fix for the upgrade scenario here is to fix more of the small issues in the lib-bucket-provisioner and make it so it will overwrite existing secrets and configmaps if they already exist rather than giving up and erroring out.