K8SSAND-1853 ⁃ Support additional jvm options for the various option files

[adejanovski]

Cassandra uses multiple jvm options files (jvm.options, jvm-server.options, jvm8-server.options, jvm11-server.options, cassandra-env.sh), and we're currently allowing to pass unstructured options to cassandra-env.sh only.

We'd need to have new []string additional jvm options fields for each file in the CassandraConfig.JvmOptions struct:

config:
  jvmOptions:
      additionalJvmServerOptions:
        - '-XX:+UseConcMarkSweepGC'
        - '-XX:+UseCMSInitiatingOccupancyOnly'
        - '-XX:+CMSParallelInitialMarkEnabled'
      additionalJvm8ServerOptions:
        - '-XX:ThreadPriorityPolicy=42'
      additionalJvm11ServerOptions:
        - '-Djdk.attach.allowAttachSelf=true'

would map in the cassdc to:

  config:
    jvm-server-options:
      additional-jvm-opts:
        - '-XX:+UseConcMarkSweepGC'
        - '-XX:+UseCMSInitiatingOccupancyOnly'
        - '-XX:+CMSParallelInitialMarkEnabled'
    jvm11-server-options:
      additional-jvm-opts:
        - '-Djdk.attach.allowAttachSelf=true'
    jvm8-server-options:
      additional-jvm-opts:
        - '-XX:ThreadPriorityPolicy=42'

Additional options for the cassandra-env.sh would still be handled by the  JvmOptions.AdditionalOptions field as they currently are.

┆Issue is synchronized with this Jira Task by Unito
┆friendlyId: K8SSAND-1853
┆priority: Medium

[adutra]

Currently in k8ssandra-operator, JVM options are exposed as a high-level, schemaful struct. The operator assigns each user-provided value to the appropriate cass-config-builder key, so the whole complexity around having multiple JVM options files is hidden from the user.

What you are asking would require to go schemaless for JVM options, pretty much what we did for cassandra.yaml and dse.yaml recently.

[adejanovski]

What you are asking would require to go schemaless for JVM options, pretty much what we did for cassandra.yaml and dse.yaml recently.

I'd say it's a little different since we're trying to assign a list of strings to separate options files, whereas the cassandra and dse yaml files are structured as a map[string]interface{} type, all going to the same file.

Based on our offline conversation, what do you think of the following structure?

      - config:
          jvmOptions:
            additionalJvm11ServerOptions:
              - '-XX:+UseConcMarkSweepGC'
              - '-XX:+CMSParallelRemarkEnabled'
              - '-XX:+UseCMSInitiatingOccupancyOnly'
            additionalJvm8ServerOptions:
              - '-XX:ThreadPriorityPolicy=42'
            additionalJvmServerOptions:
              - '-Dio.netty.maxDirectMemory=0'

[adutra]

Yes, after our conversation, I agree that your suggestion is a good compromise. 👍