Investigate Helm charts for individual resources managed by the ECK operator

[pebrc]

We currently have a Helm chart for the operator itself. However we do not have Helm charts that allow users to install individual Elastic Stack applications like Elasticsearch, Kibana etc through Helm.

Historically we have been reluctant to built such a chart because we are afraid we will have to more or less recreate the full CRDs through Helm templating.

Given that Helm charts are fairly easy to create we would like to create a few prototypes of different approaches to this problem so that we can make a more informed decision on how to proceed.

Specifically it would be good to draft maybe one or two versions of a Helm chart, maybe starting with Elasticsearch which is the most complex application managed by ECK:

• a very simple wrapper around the CRD with templating just for the version and the name as these are standardised in Helm
• a more opinionated approach with additional features (maybe with virtual memory setup, file realm users, more complex node topology)

An additional question to answer would be how we would maintain the Helm chart as we release new versions of the operator. For example is there an easy way to generate it from the latest CRD version by applying a few textual transformations. How big would the maintenance effort be if we do not do that.

[noslowerdna]

@pebrc I was looking for this exact feature (specifically, a Helm chart wrapping the Elasticsearch CRD) a few months ago, and didn't find anything apart from a blog post recommending that approach ("Elastic on Elastic: How InfoSec deploys infrastructure and stays up-to-date with ECK") and an old extremely generic project, eck-helm-chart. If it helps to have as a starting point or standalone example, here's a gist with the simple Helm chart I ended up creating to meet our needs (any Helm style feedback is welcome as I don't have much experience in this space). It's rather opinionated and somewhat customized for our environment, so I'm sure substantial adjustments would be necessary.

Since there are so many disparate and possibly complicated options, perhaps a curated library of various examples accumulated from the community along with accompanying documentation providing technical guidance and consensus best practices would be a reasonable solution to satisfy the objectives of this issue. That could get most people 80% of the way there; I know it would have been very helpful in my case.

[kaykhancheckpoint]

Any update on this?

[pebrc]

First draft of how we could structure the values.yaml file, comments inline:

version: 8.2.0

# Initially I thought we should just expose raw nodeSets through the values file. But now I think they could be used as
# a sort of escape hatch for users for whom the templating below is too rigid
nodeSets:
  - name: default
    config:
      node.roles: ["master", "data", "ingest"]
    podTemplate:
      spec:
        containers:
          - name: elasticsearch
            resources:
              limits:
                memory: 8Gi
                cpu: 1


# ... because the preferred way to use the Helm chart would be one of the predefined topologies

# choose from: mixed, dedicated masters, data_hot, data_cold, data_frozen, data_content, ml, ingest
topologies:
  - "master"
  - "data_hot"
  - "data_cold"

master:
  count: 3
  affinity: { }
  tolerations: {}
  zoneAware: false
  resources:
    limits:
      memory: 4Gi
      cpu: 1
    requests:
      storage: 10Gi
  storageClassName: enough-io

data_hot:
  count: 5
  affinity: {}
  tolerations: {}
  zoneAware: true # auto-configures https://www.elastic.co/guide/en/cloud-on-k8s/2.2/k8s-advanced-node-scheduling.html#k8s-availability-zone-awareness
  resources:
    limits:
      memory: 16Gi
      cpu: 2
    requests:
      memory: 16Gi
      cpu: 2
      storage: 100Gi
  storageClassName: high-io



serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: ""

# Shared between all node sets etc. (but not applied on raw nodeSets)

labels: {}

annotations: {}

podSecurityContext: {}
  # fsGroup: 1000

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 1000

secureSettings: [] # as in the CRD

http: {} # as in the CRD ?

transport: {} # as in the CRD

updateStrategy: {}

[pebrc]

A potential approach would be to start with raw nodeSets in a first version of this Helm chart and then extend this later with more opinionated abstractions like topologies.

A few more thoughts on the tradeoffs between the two approaches:

• nodeSets are just nodeSets including podTemplates as we have them in the CRD there would be no reduction in verbosity in their use in the Helm chart compared to the CRD
• topologies allow us to avoid repetition through templating e.g. for labels and annotations. This might especially relevant if we start including additional authentication configuration (think SAML or LDAP etc) that needs to be repeated in each nodeSet

[noslowerdna]

A potential approach would be to start with raw nodeSets in a first version of this Helm chart

Yes, that seems reasonable to me. A minimal initial version (I imagine it might be quite similar to the old eck-helm-project project that's out there?) would at least be a starting point for people who wish to use Helm for ECK without supplying their own custom chart(s). All of the information would reside in configuration values.

[jmlrt]

Based on my experience maintaining the existing Elastic Helm charts and other config management tooling relying on templates like the old ansible-elasticsearch role, I wouldn't recommend providing too much opinionated abstractions in the chart values. I think this is especially relevant for resources managed by ECK which is already providing opinionated abstractions on the Elastic product (but with a real programming language and better logic instead of tinkering with YAML templates).

By oversimplifying a lot, I think that something like that could even do the job:

• elasticsearch.yaml:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: {{ include "es-eck.fullname" . }}
  labels:
    {{- include "es-eck.labels" . | nindent 4 }}
spec:
{{ toYaml .Values.esSpec | indent 2 }}

• values.yaml:

esSpec: 
  version: 8.2.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

Some of the benefits of being very generic are:

• no need to re-invent the operator logic in the charts
• in most cases, no need to add new code to handle new features of ECK (lower maintenance for devs, users don't have to wait for a new feature implemented in ECK to be later implemented in the charts)
• no need to learn a different syntax for the chart for users
• exact same behavior between deployment with only ECK and deployment with ECK + charts (easier support, most of the issues with charts could be reproduced without chart)
• No need to try to provide custom solutions to handle every possible use case

Meanwhile, this is still providing most benefits of the Helm charts:

• charts versioning,
• Helm repository
• install, update, rollback, uninstall from a single Helm command
• compatible with all K8S deployment tools supporting Helm charts (Terraform, ArgoCD, ...)
• compatible with an umbrella chart that deploys several chart dependencies
• ...

Of course, there is a good balance to find between too much generic like the example above and too much abstraction and using raw nodeSets like in @pebrc
first example looks totally appropriate for me.