Set deviceClasses to avoid replicated pool spreading PGs across all OSDs

[malayparida2000]

BZ-https://bugzilla.redhat.com/show_bug.cgi?id=2274175

Set the deviceClasses of the pools to avoid wrong data placement

Earlier when replica-1 was getting enabled on an existing cluster the pool was setting it's deviceClass to replicated but there were no osds with this deviceClass as osd's during normal creation were given SSD deviceClass if blank, and now they can't change. To avoid & solve this problem we have to intelligently set the deviceClass for the pools, Depending on whether replica-1 is enabled & the number of deviceClasses found in the cephCluster CR status.

To do this we added a DefaultCephDeviceClass field in StorageCluster Status which will hold the default ceph device class to be used for the pools. Depending on whether replica-1 is enabled & the number of deviceClasses found in the status this value is determined and set in the status.

[malayparida2000]

The unit test failure should be fixed by this #2619

[iamniting]

/hold

[iamniting]

Approved mistakenly

[travisn]

/approve
/lgtm

[malayparida2000]

/test ocs-operator-bundle-e2e-aws

[agarwal-mudit]

/cherry-pick release-4.16

[openshift-cherrypick-robot]

@agarwal-mudit: once the present PR merges, I will cherry-pick it on top of release-4.16 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[malayparida2000]

2024-06-10T07:34:24.303690686Z time="2024-06-10T07:34:24Z" level=info msg="creating bucket nb.1718003653158.makestoragegreatagain.com" sys=openshift-storage/noobaa
2024-06-10T07:34:24.669995832Z time="2024-06-10T07:34:24Z" level=error msg="got error when trying to create bucket nb.1718003653158.makestoragegreatagain.com. error: RequestError: send request failed\ncaused by: Put \"https://s3.us-east-2.amazonaws.com/nb.1718003653158.makestoragegreatagain.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa
2024-06-10T07:34:24.669995832Z time="2024-06-10T07:34:24Z" level=info msg="SetPhase: temporary error during phase \"Configuring\"" sys=openshift-storage/noobaa
2024-06-10T07:34:24.670049595Z time="2024-06-10T07:34:24Z" level=warning msg="⏳ Temporary Error: RequestError: send request failed\ncaused by: Put \"https://s3.us-east-2.amazonaws.com/nb.1718003653158.makestoragegreatagain.com\": tls: failed to verify certificate: x509: certificate signed by unknown authority" sys=openshift-storage/noobaa
2024-06-10T07:34:24.678836665Z time="2024-06-10T07:34:24Z" level=info msg="UpdateStatus: Done generation 1" sys=openshift-storage/noobaa
2024-06-10T07:34:24.679320223Z time="2024-06-10T07:34:24Z" level=info msg="Update event detected for noobaa (openshift-storage), queuing Reconcile"

The noobaa problem keeps failing the e2e test again n again

[malayparida2000]

/retest

[malayparida2000]

/retest

[malayparida2000]

I did extensive testing for the PR with the below scenarios
Case 1: Cluster was created with old code, then upgraded to new code
Case 2: Cluster was created & replica 1 was enabled on the old code, then upgraded to new code
Case 3: Cluster was created with replica-1 with old code, then upgraded to new code
Case 4: Cluster gets created & replica-1 gets enabled both on new code
Case 5: Cluster is created from scratch with replica-1 with new code
here is the complete testing results https://hackmd.io/@Yh4a4hAATcW2BNYBJVSx4w/ryLe_4kHR

Happy to state that the code works for all the above cases & solves the bug.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: iamniting, malayparida2000, travisn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [iamniting]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-cherrypick-robot]

@agarwal-mudit: #2615 failed to apply on top of branch "release-4.16":

Applying: Set the deviceClasses of the pools to avoid wrong data placement
Using index info to reconstruct a base tree...
M	api/v1/storagecluster_types.go
M	controllers/storagecluster/cephcluster.go
M	controllers/storagecluster/cephcluster_test.go
M	controllers/storagecluster/generate.go
Falling back to patching base and 3-way merge...
Auto-merging controllers/storagecluster/generate.go
Auto-merging controllers/storagecluster/cephcluster_test.go
CONFLICT (content): Merge conflict in controllers/storagecluster/cephcluster_test.go
Auto-merging controllers/storagecluster/cephcluster.go
Auto-merging api/v1/storagecluster_types.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Set the deviceClasses of the pools to avoid wrong data placement
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.