Helm: Major refactoring regarding Deployments, Environment variables …

…and more (#20733)

* fix(chart): Remove non-functional Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler (HPA) refers to a Deployment that
doesn't exist and therefore can not work. As a result it's
pointless to carry it around in this chart and give the wrong
impression it could work. This patch removes it from the helm
chart and drops all references to it.

* refactor(chart): Refactor sidekiq deployments to scale

This patch reworks how the sidekiq deployment is set up, by
splitting it into many sidekiq deployments, but at least one,
which should allow to scale the number of sidekiq jobs as
expected while being friendly to single user instances as well
as larger ones.

Further it introduces per deployment overwrites for the most
relevant pod fields like resources, affinities and processed
queues, number of jobs and the sidekiq security contexts.

The exact implementation was inspired by an upstream issue:

https://github.com/mastodon/mastodon/issues/20453

* fix(chart): Remove linode default values from values

This patch drops the linode defaults from the values.yaml since
these are not obvious and can cause unexpected connections as
well as leaking secrets to linode, when other s3 storage
backends are used and don't explicitly configure these options
by accident.

Mastodon will then try to authenticate to the linode backends
and therefore disclose the authentication secrets.

* refactor(chart): Rework reduce value reference duplication

Since most of the values are simply setup like this:

```
{{- if .Values.someVariable }}
SOME_VARIABLE: {{ .Values.someVariable }}
{{- end }}
```

There is a lot of duplication in the references in order to
full in the variables. There is an equivalent notation, which
reduces the usage of the variable name to just once:

```
{{- with .Values.someVariable }}
SOME_VARIABLE: {{ . }}
{{- end }}
```

What seems like a pointless replacement, will reduce potential
mistakes down the line by possibly only adjusting one of the
two references.

* fix(chart): Switch to new OMNIAUTH_ONLY variable

This patch adjusts the helm chart to use the new `OMNIAUTH_ONLY`
variable, which replaced the former
`OAUTH_REDIRECT_AT_SIGN_IN` variable in the following commit:

mastodon/mastodon#17288
mastodon/mastodon@3c88579

* fix(chart): Repair connection test to existing service

Currently the connect test can't work, since it's connecting to
a non-existing service this patch fixes the service name to
make the job connect to the mastodon web service to verify the
connection.

* docs(chart): Adjust values.yaml to support helm-docs

This patch updates most values to prepare an introduction of
helm-docs. This should help to make the chart more user
friendly by explaining the variables and provide a standardised
README file, like many other helm charts do.

References:
https://github.com/norwoodj/helm-docs

* refactor(chart): Allow individual overwrites for streaming and web deployment

This patch works how the streaming and web deployments work by
adding various fields to overwrite values such as affinities,
resources, replica count, and security contexts.

BREAKING CHANGE: This commit removes `.Values.replicaCount` in
favour of `.Values.mastodon.web.replicas` and
`.Values.mastodon.streaming.values`.

* feat(chart): Add option for authorized fetch

Currently the helm chart doesn't support authorized fetch aka.
"Secure Mode" this patch fixes that by adding the needed config
option to the values file and the configmap.

* docs(chart): Improve helm-docs compatiblity

This patch adjust a few more comments in the values.yaml to be
picked up by helm-docs. This way, future adoption is properly
prepared.

* fix(chart): Add automatic detection of scheduler sidekiq queue

This patch adds an automatic switch to the `Recreate` strategy
for the sidekiq Pod in order to prevent accidental concurrency
for the scheduler queue.

* fix(chart): Repair broken DB_POOL variable

templates/_helpers.tpl

@@ -136,3 +136,15 @@ Return true if a mastodon secret object should be created
     {{- true -}}
 {{- end -}}
 {{- end -}}
+
+{{/*
+Find highest number of needed database connections to set DB_POOL variable
+*/}}
+{{- define "mastodon.maxDbPool" -}}
+{{/* Default MAX_THREADS for Puma is 5 */}}
+{{- $poolSize := 5 }}
+{{- range .Values.mastodon.sidekiq.workers }}
+{{- $poolSize = max $poolSize .concurrency }}
+{{- end }}
+{{- $poolSize | quote }}
+{{- end }}

templates/deployment-sidekiq.yaml

@@ -1,130 +1,132 @@
+{{- $context := . }}
+{{- range .Values.mastodon.sidekiq.workers }}
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: {{ include "mastodon.fullname" . }}-sidekiq
+  name: {{ include "mastodon.fullname" $context }}-sidekiq-{{ .name }}
   labels:
-    {{- include "mastodon.labels" . | nindent 4 }}
+    {{- include "mastodon.labels" $context | nindent 4 }}
+    app.kubernetes.io/component: sidekiq-{{ .name }}
+    app.kubernetes.io/part-of: rails
 spec:
-  {{- if not .Values.autoscaling.enabled }}
-  replicas: {{ .Values.replicaCount }}
+  replicas: {{ .replicas }}
+  {{- if (has "scheduler" .queues) }}
+  strategy:
+    type: Recreate
   {{- end }}
   selector:
     matchLabels:
-      {{- include "mastodon.selectorLabels" . | nindent 6 }}
-      app.kubernetes.io/component: sidekiq
+      {{- include "mastodon.selectorLabels" $context | nindent 6 }}
+      app.kubernetes.io/component: sidekiq-{{ .name }}
       app.kubernetes.io/part-of: rails
   template:
     metadata:
       annotations:
-        {{- with .Values.podAnnotations }}
+        {{- with $context.Values.podAnnotations }}
         {{- toYaml . | nindent 8 }}
         {{- end }}
         # roll the pods to pick up any db migrations or other changes
-        {{- include "mastodon.rollingPodAnnotations" . | nindent 8 }}
+        {{- include "mastodon.rollingPodAnnotations" $context | nindent 8 }}
       labels:
-        {{- include "mastodon.selectorLabels" . | nindent 8 }}
-        app.kubernetes.io/component: sidekiq
+        {{- include "mastodon.selectorLabels" $context | nindent 8 }}
+        app.kubernetes.io/component: sidekiq-{{ .name }}
         app.kubernetes.io/part-of: rails
     spec:
-      {{- with .Values.imagePullSecrets }}
+      {{- with $context.Values.imagePullSecrets }}
       imagePullSecrets:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      serviceAccountName: {{ include "mastodon.serviceAccountName" . }}
-      {{- with .Values.podSecurityContext }}
+      serviceAccountName: {{ include "mastodon.serviceAccountName" $context }}
+      {{- with (default $context.Values.podSecurityContext $context.Values.mastodon.sidekiq.podSecurityContext) }}
       securityContext:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- if (not .Values.mastodon.s3.enabled) }}
-      # ensure we run on the same node as the other rails components; only
-      # required when using PVCs that are ReadWriteOnce
-      {{- if or (eq "ReadWriteOnce" .Values.mastodon.persistence.assets.accessMode) (eq "ReadWriteOnce" .Values.mastodon.persistence.system.accessMode) }}
+      {{- with (default (default $context.Values.affinity $context.Values.mastodon.sidekiq.affinity) .affinity) }}
       affinity:
-        podAffinity:
-          requiredDuringSchedulingIgnoredDuringExecution:
-            - labelSelector:
-                matchExpressions:
-                  - key: app.kubernetes.io/part-of
-                    operator: In
-                    values:
-                      - rails
-              topologyKey: kubernetes.io/hostname
+        {{- toYaml . | nindent 8 }}
       {{- end }}
+      {{- if (not $context.Values.mastodon.s3.enabled) }}
       volumes:
         - name: assets
           persistentVolumeClaim:
-            claimName: {{ template "mastodon.fullname" . }}-assets
+            claimName: {{ template "mastodon.fullname" $context }}-assets
         - name: system
           persistentVolumeClaim:
-            claimName: {{ template "mastodon.fullname" . }}-system
+            claimName: {{ template "mastodon.fullname" $context }}-system
       {{- end }}
       containers:
-        - name: {{ .Chart.Name }}
+        - name: {{ $context.Chart.Name }}
           securityContext:
-            {{- toYaml .Values.securityContext | nindent 12 }}
-          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
-          imagePullPolicy: {{ .Values.image.pullPolicy }}
+            {{- toYaml $context.Values.mastodon.sidekiq.securityContext | nindent 12 }}
+          image: "{{ $context.Values.image.repository }}:{{ $context.Values.image.tag | default $context.Chart.AppVersion }}"
+          imagePullPolicy: {{ $context.Values.image.pullPolicy }}
           command:
             - bundle
             - exec
             - sidekiq
             - -c
-            - {{ .Values.mastodon.sidekiq.concurrency | quote }}
+            - {{ .concurrency | quote }}
+            {{- range .queues }}
+            - -q
+            - {{ . | quote }}
+            {{- end }}
           envFrom:
             - configMapRef:
-                name: {{ include "mastodon.fullname" . }}-env
+                name: {{ include "mastodon.fullname" $context }}-env
             - secretRef:
-                name: {{ template "mastodon.secretName" . }}
+                name: {{ template "mastodon.secretName" $context }}
           env:
             - name: "DB_PASS"
               valueFrom:
                 secretKeyRef:
-                  name: {{ template "mastodon.postgresql.secretName" . }}
+                  name: {{ template "mastodon.postgresql.secretName" $context }}
                   key: password
             - name: "REDIS_PASSWORD"
               valueFrom:
                 secretKeyRef:
-                  name: {{ template "mastodon.redis.secretName" . }}
+                  name: {{ template "mastodon.redis.secretName" $context }}
                   key: redis-password
-            {{- if (and .Values.mastodon.s3.enabled .Values.mastodon.s3.existingSecret) }}
+            {{- if (and $context.Values.mastodon.s3.enabled $context.Values.mastodon.s3.existingSecret) }}
             - name: "AWS_SECRET_ACCESS_KEY"
               valueFrom:
                 secretKeyRef:
-                  name: {{ .Values.mastodon.s3.existingSecret }}
+                  name: {{ $context.Values.mastodon.s3.existingSecret }}
                   key: AWS_SECRET_ACCESS_KEY
             - name: "AWS_ACCESS_KEY_ID"
               valueFrom:
                 secretKeyRef:
                   name: {{ .Values.mastodon.s3.existingSecret }}
                   key: AWS_ACCESS_KEY_ID
             {{- end }}
-            {{- if .Values.mastodon.smtp.existingSecret }}
+            {{- if $context.Values.mastodon.smtp.existingSecret }}
             - name: "SMTP_LOGIN"
               valueFrom:
                 secretKeyRef:
-                  name: {{ .Values.mastodon.smtp.existingSecret }}
+                  name: {{ $context.Values.mastodon.smtp.existingSecret }}
                   key: login
                   optional: true
             - name: "SMTP_PASSWORD"
               valueFrom:
                 secretKeyRef:
-                  name: {{ .Values.mastodon.smtp.existingSecret }}
+                  name: {{ $context.Values.mastodon.smtp.existingSecret }}
                   key: password
             {{- end }}
-          {{- if (not .Values.mastodon.s3.enabled) }}
+          {{- if (not $context.Values.mastodon.s3.enabled) }}
           volumeMounts:
             - name: assets
               mountPath: /opt/mastodon/public/assets
             - name: system
               mountPath: /opt/mastodon/public/system
           {{- end }}
           resources:
-            {{- toYaml .Values.resources | nindent 12 }}
-      {{- with .Values.nodeSelector }}
+            {{- toYaml (default (default $context.Values.resources $context.Values.mastodon.sidekiq.resources) .resources) | nindent 12 }}
+      {{- with $context.Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.tolerations }}
+      {{- with $context.Values.tolerations }}
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+{{- end }}

templates/deployment-streaming.yaml

@@ -5,17 +5,15 @@ metadata:
   labels:
     {{- include "mastodon.labels" . | nindent 4 }}
 spec:
-  {{- if not .Values.autoscaling.enabled }}
-  replicas: {{ .Values.replicaCount }}
-  {{- end }}
+  replicas: {{ .Values.mastodon.streaming.replicas }}
   selector:
     matchLabels:
       {{- include "mastodon.selectorLabels" . | nindent 6 }}
       app.kubernetes.io/component: streaming
   template:
     metadata:
       annotations:
-        {{- with .Values.podAnnotations }}
+        {{- with (default .Values.podAnnotations .Values.mastodon.streaming.podAnnotations) }}
         {{- toYaml . | nindent 8 }}
         {{- end }}
         # roll the pods to pick up any db migrations or other changes
@@ -29,13 +27,13 @@ spec:
         {{- toYaml . | nindent 8 }}
       {{- end }}
       serviceAccountName: {{ include "mastodon.serviceAccountName" . }}
-      {{- with .Values.podSecurityContext }}
+      {{- with (default .Values.podSecurityContext .Values.mastodon.streaming.podSecurityContext) }}
       securityContext:
         {{- toYaml . | nindent 8 }}
       {{- end }}
       containers:
-        - name: {{ .Chart.Name }}
-          {{- with .Values.securityContext }}
+        - name: {{ .Chart.Name }}-streaming
+          {{- with (default .Values.securityContext .Values.mastodon.streaming.securityContext) }}
           securityContext:
             {{- toYaml . | nindent 12 }}
           {{- end }}
@@ -72,15 +70,15 @@ spec:
             httpGet:
               path: /api/v1/streaming/health
               port: streaming
-          {{- with .Values.resources }}
+          {{- with (default .Values.resources .Values.mastodon.streaming.resources) }}
           resources:
             {{- toYaml . | nindent 12 }}
           {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.affinity }}
+      {{- with (default .Values.affinity .Values.mastodon.streaming.affinity) }}
       affinity:
         {{- toYaml . | nindent 8 }}
       {{- end }}

templates/deployment-web.yaml

@@ -5,9 +5,7 @@ metadata:
   labels:
     {{- include "mastodon.labels" . | nindent 4 }}
 spec:
-  {{- if not .Values.autoscaling.enabled }}
-  replicas: {{ .Values.replicaCount }}
-  {{- end }}
+  replicas: {{ .Values.mastodon.web.replicas }}
   selector:
     matchLabels:
       {{- include "mastodon.selectorLabels" . | nindent 6 }}
@@ -16,7 +14,7 @@ spec:
   template:
     metadata:
       annotations:
-        {{- with .Values.podAnnotations }}
+        {{- with (default .Values.podAnnotations .Values.mastodon.web.podAnnotations) }}
         {{- toYaml . | nindent 8 }}
         {{- end }}
         # roll the pods to pick up any db migrations or other changes
@@ -31,7 +29,7 @@ spec:
         {{- toYaml . | nindent 8 }}
       {{- end }}
       serviceAccountName: {{ include "mastodon.serviceAccountName" . }}
-      {{- with .Values.podSecurityContext }}
+      {{- with (default .Values.podSecurityContext .Values.mastodon.web.podSecurityContext) }}
       securityContext:
         {{- toYaml . | nindent 8 }}
       {{- end }}
@@ -45,8 +43,8 @@ spec:
             claimName: {{ template "mastodon.fullname" . }}-system
       {{- end }}
       containers:
-        - name: {{ .Chart.Name }}
-          {{- with .Values.securityContext }}
+        - name: {{ .Chart.Name }}-web
+          {{- with (default .Values.securityContext .Values.mastodon.web.securityContext) }}
           securityContext:
             {{- toYaml . | nindent 12 }}
           {{- end }}
@@ -112,15 +110,15 @@ spec:
               port: http
             failureThreshold: 30
             periodSeconds: 5
-          {{- with .Values.resources }}
+          {{- with (default .Values.resources .Values.mastodon.web.resources) }}
           resources:
             {{- toYaml . | nindent 12 }}
           {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.affinity }}
+      {{- with (default .Values.affinity .Values.mastodon.web.affinity) }}
       affinity:
         {{- toYaml . | nindent 8 }}
       {{- end }}

templates/tests/test-connection.yaml

@@ -11,5 +11,5 @@ spec:
     - name: wget
       image: busybox
       command: ['wget']
-      args: ['{{ include "mastodon.fullname" . }}:{{ .Values.service.port }}']
+      args: ['{{ include "mastodon.fullname" . }}-web:{{ .Values.service.port }}']
   restartPolicy: Never