Helm: Simplified TLS configuration

[bradleypettit]

What this PR does

This PR adds a new tls key to the Helm values, and updated templates to simplify the configuration of TLS between Mimir services.

This change is for traffic within the Mimir/GEM cluster, including Memcached (i.e. not how the cluster is exposed to clients).

Currently, enabling encryption in transit within the cluster requires a lot of configuration changes. Below is an example of some of the changes required for a GEM cluster (there are other values, but I've kept only the settings required to enable TLS).

Current values required for enabling TLS

---
tls_config_block: &tls-config-block
  tls_enabled: true
  tls_cert_path: /certificates/server.crt
  tls_key_path: /certificates/server.key
  tls_ca_path: /certificates/root.crt
  tls_insecure_skip_verify: true

global:
  extraVolumes:
    - name: cert-files
      secret:
        secretName: gem-certs
  extraVolumeMounts:
    - mountPath: /certificates/
      name: cert-files
      readOnly: false

mimir:
  structuredConfig:
    server:
      http_tls_config:
        cert_file: /certificates/server.crt
        key_file: /certificates/server.key
        client_auth_type: RequestClientCert
        client_ca_file: /certificates/root.crt
      grpc_tls_config:
        cert_file: /certificates/server.crt
        key_file: /certificates/server.key
        client_auth_type: RequestClientCert
        client_ca_file: /certificates/root.crt
    admin_api:
      leader_election:
        client_config: *tls-config-block
    admin_client:
      storage:
        cache:
          memcached: *tls-config-block
    blocks_storage:
      bucket_store:
        index_cache:
          memcached: *tls-config-block
        chunks_cache:
          memcached: *tls-config-block
        metadata_cache:
          memcached: *tls-config-block
    ruler_storage:
      cache:
        memcached: *tls-config-block
    instrumentation:
      distributor_client: *tls-config-block
    gateway:
      proxy:
        admin_api:
          <<: *tls-config-block
          url: https://{{ template "mimir.fullname" . }}-admin-api.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        alertmanager:
          <<: *tls-config-block
          url: https://{{ template "mimir.fullname" . }}-alertmanager-headless.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        compactor:
          <<: *tls-config-block
          url: https://{{ template "mimir.fullname" . }}-compactor.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        default:
          <<: *tls-config-block
          url: https://{{ template "mimir.fullname" . }}-admin-api.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        distributor:
          <<: *tls-config-block
          url: https://{{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        ingester:
          <<: *tls-config-block
          url: https://{{ template "mimir.fullname" . }}-ingester-headless.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        query_frontend:
          <<: *tls-config-block
          url: https://{{ template "mimir.fullname" . }}-query-frontend.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        ruler:
          <<: *tls-config-block
          url: https://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        store_gateway:
          <<: *tls-config-block
          url: https://{{ template "mimir.fullname" . }}-store-gateway-headless.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
    query_scheduler:
      grpc_client_config: *tls-config-block
    frontend_worker:
      grpc_client_config: *tls-config-block
      query_scheduler_grpc_client_config: *tls-config-block
    ingester_client:
      grpc_client_config: *tls-config-block
    frontend:
      grpc_client_config: *tls-config-block
      results_cache:
        memcached: *tls-config-block
    querier:
      store_gateway_client: *tls-config-block
    ruler:
      alertmanager_url: https://{{ template "mimir.fullname" . }}-alertmanager-headless.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}/alertmanager
      alertmanager_client:
        <<: *tls-config-block
      ruler_client: *tls-config-block
      query_frontend:
        grpc_client_config: *tls-config-block
    alertmanager:
      alertmanager_client: *tls-config-block

tls-readiness-probe: &tls-readiness-probe
  readinessProbe:
    httpGet:
      scheme: HTTPS

admin_api: *tls-readiness-probe
alertmanager: *tls-readiness-probe
compactor: *tls-readiness-probe
distributor: *tls-readiness-probe
gateway: 
  <<: *tls-readiness-probe
ingester: 
  <<: *tls-readiness-probe
overrides_exporter:
  <<: *tls-readiness-probe
  livenessProbe:
    httpGet:
      scheme: HTTPS
querier: *tls-readiness-probe
query_frontend: *tls-readiness-probe
query_scheduler: *tls-readiness-probe
ruler:
  <<: *tls-readiness-probe
ruler_querier: *tls-readiness-probe
ruler_query_frontend: *tls-readiness-probe
ruler_query_scheduler: *tls-readiness-probe
store_gateway: *tls-readiness-probe

memcache-modifications: &memcache-modifications
  enabled: true
  extraVolumes:
    - name: memcached-exporter-config
      configMap:
        name: gem-memcached-exporter-config
        items:
        - key: web-config.yaml
          path: web-config.yaml
  extraVolumeMounts:
    - mountPath: /etc/memcached_exporter
      name: memcached-exporter-config
  extraExtendedOptions: "ssl_chain_cert=/certificates/server.crt,ssl_key=/certificates/server.key"
  extraArgs:
    Z:

admin-cache: *memcache-modifications
index-cache: *memcache-modifications
chunks-cache: *memcache-modifications
metadata-cache: *memcache-modifications
results-cache: *memcache-modifications

memcachedExporter:
  extraArgs:
    memcached.tls.enable:
    memcached.tls.cert-file: /certificates/server.crt
    memcached.tls.key-file: /certificates/server.key
    memcached.tls.ca-file: /certificates/root.crt
    memcached.tls.insecure-skip-verify:
    web.config.file: /etc/memcached_exporter/web-config.yaml

With the changes to templates and new default values provided in this PR, the values file is much simpler and becomes (for example).

Proposed values required for enabling TLS

---
tls:
  mimir:
    enabled: true
    server:
      client_auth_type: RequestClientCert
    client:
      server_name: gem-cluster-name-san
  memcached:
    enabled: true
    client:
      server_name: gem-cluster-name-san

global:
  extraVolumes:
    - name: cert-files
      secret:
        secretName: gem-certs
  extraVolumeMounts:
    - mountPath: /etc/mimir/certificates/
      name: cert-files
      readOnly: true

Another comparison is available in the 2 CI tests: enterprise-https-values.yaml (original) and enterprise-https-simple-values.yaml (using the new tls object).

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]. If changelog entry is not needed, please add the changelog-not-needed label to the PR.
• about-versioning.md updated with experimental features.

[bradleypettit]

Mindful that I haven't got any docs in this PR yet - definitely will add some examples of how to use the new values that are available to people. Possibly as an update to [Securing Communications with TLS].(https://grafana.com/docs/mimir/latest/manage/secure/securing-communications-with-tls/)

Just holding off on that temporarily until the team has had a chance to review, in case any substantial changes are needed.