Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/redis-ha/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ keywords:
- redis
- keyvalue
- database
version: 4.34.6
version: 4.34.7
appVersion: 8.2.1
description: This Helm chart provides a highly available Redis implementation with a master/slave configuration and uses Sentinel sidecars for failover management
icon: https://img.icons8.com/external-tal-revivo-shadow-tal-revivo/24/external-redis-an-in-memory-data-structure-project-implementing-a-distributed-logo-shadow-tal-revivo.png
Expand Down
11 changes: 8 additions & 3 deletions charts/redis-ha/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ The following table lists the configurable parameters of the Redis chart and the
| `affinity` | Override all other affinity settings for the Redis server pods with a string. | string | `""` |
| `auth` | Configures redis with AUTH (requirepass & masterauth conf params) | bool | `false` |
| `authKey` | Defines the key holding the redis password in existing secret. | string | `"auth"` |
| `authSecretAnnotations` | Annotations for auth secret | object | `{}` |
| `configmap.labels` | Custom labels for the redis configmap | object | `{}` |
| `configmapTest.image` | Image for redis-ha-configmap-test hook | object | `{"repository":"koalaman/shellcheck","tag":"v0.10.0"}` |
| `configmapTest.image.repository` | Repository of the configmap shellcheck test image. | string | `"koalaman/shellcheck"` |
Expand Down Expand Up @@ -173,6 +174,7 @@ The following table lists the configurable parameters of the Redis chart and the
| `serviceLabels` | Custom labels for redis service | object | `{}` |
| `splitBrainDetection.interval` | Interval between redis sentinel and server split brain checks (in seconds) | int | `60` |
| `splitBrainDetection.resources` | splitBrainDetection resources | object | `{}` |
| `splitBrainDetection.retryInterval` | | int | `10` |
| `sysctlImage.command` | sysctlImage command to execute | list | `[]` |
| `sysctlImage.enabled` | Enable an init container to modify Kernel settings | bool | `false` |
| `sysctlImage.mountHostSys` | Mount the host `/sys` folder to `/host-sys` | bool | `false` |
Expand Down Expand Up @@ -461,15 +463,18 @@ Should your Pod require additional egress rules, define them in a `egressRules`

## Sentinel and redis server split brain detection

Under not entirely known yet circumstances redis sentinel and its corresponding redis server reach a condition that this chart authors call "split brain" (for short). The observed behaviour is the following: the sentinel switches to the new re-elected master, but does not switch its redis server. Majority of original discussion on the problem has happened at the <https://github.com/DandyDeveloper/charts/issues/121>.
Under not entirely known yet circumstances redis sentinel and its corresponding redis server reach a condition that this chart authors call "split brain" (for short). The observed behaviour is the following: the sentinel switches to the new re-elected master, but does not switch its redis server. Majority of original discussion on the problem has happened at the #121.

The proposed solution is currently implemented as a sidecar container that runs a bash script with the following logic:
1. At intervals defined by `splitBrainDetection.interval`, the sidecar checks which node is recognized as master by Sentinel.

1. At intervals defined by splitBrainDetection.interval, the sidecar checks which node is recognized as master by Sentinel.
2. If the current pod is the master according to Sentinel, it verifies that the local Redis server is also running as master.
3. If the current pod is not the master, it ensures the local Redis server is replicating from the correct master node.
4. If any of these checks fail, the sidecar will retry the check at intervals defined by `splitBrainDetection.retryInterval`.
4. If any of these checks fail, the sidecar will retry the check at intervals defined by splitBrainDetection.retryInterval.
5. If the checks continue to fail after the retry attempts, the sidecar triggers a reinitialization: it regenerates the Redis configuration and instructs the Redis server to shut down. Kubernetes will then automatically restart the container.

# Change Log

## 4.14.9 - ** POTENTIAL BREAKING CHANGE. **
Introduced the ability to change the Haproxy Deployment container pod
- Container port in redis-haproxy-deployment.yam has been changed. Was **redis.port** To **haproxy.containerPort**. Default value is 6379.
Expand Down
12 changes: 6 additions & 6 deletions charts/redis-ha/README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -214,15 +214,15 @@ Should your Pod require additional egress rules, define them in a `egressRules`

## Sentinel and redis server split brain detection

Under not entirely known yet circumstances redis sentinel and its corresponding redis server reach a condition that this chart authors call "split brain" (for short). The observed behaviour is the following: the sentinel switches to the new re-elected master, but does not switch its redis server. Majority of original discussion on the problem has happened at the <https://github.com/DandyDeveloper/charts/issues/121>.
Under not entirely known yet circumstances redis sentinel and its corresponding redis server reach a condition that this chart authors call "split brain" (for short). The observed behaviour is the following: the sentinel switches to the new re-elected master, but does not switch its redis server. Majority of original discussion on the problem has happened at the #121.

The proposed solution is currently implemented as a sidecar container that runs a bash script with the following logic:

1. Every `splitBrainDetection.interval` seconds a master (as known by sentinel) is determined
1. If it is the current node: ensure the redis server's role is master as well.
1. If it is not the current node: ensure the redis server also replicates from the same node.

If any of the checks above fails - the redis server reinitialisation happens (it regenerates configs the same way it's done during the pod init), and then the redis server is instructed to shutdown. Then kubernetes restarts the container immediately.
1. At intervals defined by splitBrainDetection.interval, the sidecar checks which node is recognized as master by Sentinel.
2. If the current pod is the master according to Sentinel, it verifies that the local Redis server is also running as master.
3. If the current pod is not the master, it ensures the local Redis server is replicating from the correct master node.
4. If any of these checks fail, the sidecar will retry the check at intervals defined by splitBrainDetection.retryInterval.
5. If the checks continue to fail after the retry attempts, the sidecar triggers a reinitialization: it regenerates the Redis configuration and instructs the Redis server to shut down. Kubernetes will then automatically restart the container.


# Change Log
Expand Down
4 changes: 4 additions & 0 deletions charts/redis-ha/templates/redis-auth-secret.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
{{- if and .Values.auth (not .Values.existingSecret) -}}
{{ $fullAnnotations := mustMerge .Values.redis.annotations .Values.authSecretAnnotations }}
apiVersion: v1
kind: Secret
metadata:
name: {{ template "redis-ha.fullname" . }}
namespace: {{ .Release.Namespace | quote }}
{{- with $fullAnnotations }}
annotations: {{ . | toYaml | nindent 4 }}
{{- end }}
labels:
{{ include "labels.standard" . | indent 4 }}
{{- range $key, $value := .Values.extraLabels }}
Expand Down
4 changes: 4 additions & 0 deletions charts/redis-ha/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -819,9 +819,13 @@ podDisruptionBudget: {}

# -- Configures redis with AUTH (requirepass & masterauth conf params)
auth: false

# -- (string) A password that configures a `requirepass` and `masterauth` in the conf parameters (Requires `auth: enabled`)
redisPassword: ~

# -- Annotations for auth secret
authSecretAnnotations: {}

## Use existing secret containing key `authKey` (ignores redisPassword)
## Can also store AWS S3 or SSH secrets in this secret
# -- An existing secret containing a key defined by `authKey` that configures `requirepass` and `masterauth` in the conf
Expand Down
Loading