Adding support for rgw and external-rgw in external-cluster

[parth-gr]

I am having concerns about the number of external permutations that exist for CephCluster and CephObjectStore.

With code released to date, I believe we have 3 operating permutations:

1. Local CephCluster with Local CephObjectStore
2. External-mode CephCluster with Local CephObjectStore (mons and osds are remote, but we run the RGW inside k8s) (doc)
3. External-mode CephCluster with external-mode CephObjectStore

I think all 3 of these operating permutations make sense. But I think that this PR may be opening up a 4th operating permutation, and I'm not sure this 4th operating mode makes sense as a supportable configuration:
4. Local CephCluster with external-mode CephObjectStore

Originally posted by @BlaineEXE in #10425 (comment)

We need to investigate what all works for external mode for rgw.

[BlaineEXE]

Thanks for opening this, Parth.

In addition to the investigation, I think something that would be good to do is to:

1. Decide whether we want to continue supporting permutation 2
   • plan and announce the breaking deprecation if we drop support
   • we could officially say this is unsupported but try to keep existing support on a reasonable-effort basis for now
2. write CI tests for the supported external operating modes
   • we already test permutation 1, but it would be good for us to test permutation 3 as well
   • it would be good to also test permutation 3 for RBD and CephFS as well
   • if we continue to support permutation 2, tests for that would be good, but not as critical as permutation 3