Replace secrets refs for the Cassandra pods with the mutating webhook

[adejanovski]

Remove the secret mounts for Cassandra containers and replace them with injected secrets by the webhook.

Depends upon

Beta Give feedback

1. Ability to read required secrets from other sources (file) than Kubernetes cass-operator#520
   enhancement ready +2

┆Issue is synchronized with this Jira Task by Unito
┆friendlyId: K8SSAND-1620
┆priority: Medium

[burmanm]

For background, cass-operator needs to support non-webhook installations also. And with my thinking, the multiple different secrets might have different injections (such as certificates from cert-manager and user information from Vault etc). Also, each user would probably need data from different vault secret or we might have different standards how the information should be mounted.

I'd like to see structure that we could follow in each application, that would allow defining different places where to get the information from. This example (not necessarily the exact structure we end up with, but an example) would allow defining a secretName, annotations or CSI information:

  superuser:
    annotations:
      c: b
    secretName: 
    mountPath:
    csi:
      driver: ..
  keystore:
    annotations:
      a: b

We need to define the same properties for all possible secrets that we might need to use. If annotations is set, the injection framework is assumed.

The mountPath is there, because in some cases the injection framework might be unable to inject it to our wanted destination. I think Vault at least wants them under /vault/secrets directory. So we need to know where it mounted the secret information. This can be omitted if we believe the annotations processor must be able to mount it to the destination we define.

[adejanovski]

Thanks for the update, I totally missed the fact that webhook could be prohibited in some clusters.

The mountPath is there, because in some cases the injection framework might be unable to inject it to our wanted destination. I think Vault at least wants them under /vault/secrets directory. So we need to know where it mounted the secret information. This can be omitted if we believe the annotations processor must be able to mount it to the destination we define.

That's probably not necessary, at least with Vault. It has an annotation for that.
Let's keep it for now and see during implementation if it's relevant or not.

[jsanda]

How do we deal with the webhook being prohibited in some clusters?

[adejanovski]

By mounting the k8s secrets directly like we currently do. This kind of environment cannot be supported for Vault for example, unless we enable the use of the CSI driver.