Attempting to disable internode encryption on a running cluster without downtime is not possible.

[bradfordcp]

What happened?

I was attempting to disable internode encryption on a running cluster with serverEncryptionStores specified. From the starting state of a cluster up and running with encryption enabled I:

1. Set optional: true and internode_encryption: all in cassandra.yaml. This triggered a rolling restart
2. Set optional: true and internode_encryption: none. This triggered a rolling restart which got stuck.

Specifying serverEncryptionStores doesn't necessarily result in secrets being mounted to pods instead the operator checks to see if internode_encryption is not none. In my case I wanted to keep the secrets mounted (and keystore / truststore available) while all of the nodes performed rolling restarts to the point where internode encryption was disabled.

Did you expect to see something different?

Yes, I expect that setting serverEncryptionStores and clientEncryptionSrores should populate the secrets within the pods and set the following in cassandra.yaml.

keystore: /mnt/server-keystore/keystore
keystore_password: REDACTED
truststore: /mnt/server-truststore/truststore
truststore_password: REDACTED

My next step after step 2 above was to remove the cassandra.yaml settings (optional and internode_encryption) as well as the serverEncryptionStores block.

How to reproduce it (as minimally and precisely as possible):

Environment

• K8ssandra Operator version: cr.k8ssandra.io/k8ssandra/k8ssandra-operator:5e45f078

• Kubernetes version information:

kubectl version
Client Version: v1.28.10
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.9-gke.1000000

• Kubernetes cluster kind:

GKE

• Manifests:

...
    auth: false
    cassandra:
      serverVersion: 4.0.11
      serverType: cassandra
      serverEncryptionStores:
        keystoreSecretRef:
          name: my-inode-ks
          key: keystore.jks
        truststoreSecretRef:
          name: my-inode-ks
          key: truststore.jks
        keystorePasswordSecretRef:
          name: my-inode-ks
          key: keystore-password
        truststorePasswordSecretRef:
          name: my-inode-ks
          key: truststore-password
      config:
        cassandraYaml:
          dynamic_snitch: false
          server_encryption_options:
            internode_encryption: none
            optional: true
        jvmOptions:
          heapSize: 31G
      resources:
        requests:
          cpu: 30000m
          memory: 96Gi
      datacenters:
        - metadata:
            name: dc1
          stopped: false
          size: 3
          racks:
            - name: rack1
            - name: rack2
            - name: rack3

Anything else we need to know?:

You are awesome 🚀

┆Issue is synchronized with this Jira Story by Unito

[bradfordcp]

I think the checks at

k8ssandra-operator/pkg/cassandra/encryption.go

Line 167 in 87bed5f

return enabled == true

and

k8ssandra-operator/pkg/cassandra/encryption.go

Line 177 in 87bed5f

return internodeEncryption != nil && internodeEncryption != "none"

should be removed. If we have everything needed for a keystore and truststore (IE mount points and passwords) the secrets should be mounted and appropriate values added to cassandra.yaml.

At this point I think we should let the user set (or omit!) values for internode_encryption and optional in the cassandraYaml block. Future features can handle the graceful enabling and disabling (via rolling restarts or certificate reloading depending on the serverType and serverVersion).