Updating a medusaConfiguration referenced secret should propagate

[adejanovski]

When using a medusaConfigurationRef, the original secret is copied and then replicated to the contexts and namespaces involved.
But in case the secret is updated due to credentials rotation for example, the copy doesn't get updated, which prevents from replicating the changes.

We need to make sure such secret updates get picked up by the operator and that the secret copy gets refreshed so that all medusa containers can get the new credentials.

Definition of Done

Beta Give feedback

1. Rotating a medusaConfiguration secret propagates to all clusters using the configuration

[Miles-Garnsey]

Having taken a quick look, it appears that this design is a little odd, because it is replicating secrets between from one namespace to another without using the regular replicatedSecrets types and mechanisms.

I think that the logical way to remedy this problem is to drop the current secret creation mechanism in favour of the creation of a replicatedSecret within the origin namespace of the MedusaConfig.

I think i should mention that the behaviour of this logic is likely to be undefined or generally weird in the case of namespace-scoped deployments, and mention again that we should probably remove support for namespace scoping of this operator.

[burmanm]

and mention again that we should probably remove support for namespace scoping of this operator.

That would probably prevent many users from installing the operator at all as they do not have cluster-wide access.

[Miles-Garnsey]

Having discussed this last night, it appears that our preferred option is to prevent the use of namespace-remote MedusaConfigurations within the K8ssandraCluster going forward. PR 1267 gives effect to that change.

Having implemented that change, we need to consider two scenarios:

1. The future state should rely on simply using the original secret referenced in the MedusaConfiguration. No replication is required here.
2. However, clusters created in the old world, where a MedusaConfiguration in a remote namespace was allowed need to still function. To serve these (and fix the proximate issue where copies of the bucket secret were not updated), I am making a replacement of the existing functionality in this PR so that a ReplicatedSecret will be used going forward. This will only be used for those legacy clusters which have a remote-ns MedusaConfig. Otherwise the convention will be to pick up the original secret for downstream usage.

I have a PR almost ready to go for this too.