redis-cluster failover connection issue #1787
Unanswered
musab-acharath
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Body:
When a follower pod is deleted or restarted in a RedisCluster setup, the node enters slave,fail state. The operator detects the issue but never automatically triggers CLUSTER REPLICATE to reattach the replica, resulting in an infinite reconcile loop with no self-healing.
Environment
Operator: OT-CONTAINER-KIT redis-operator (OperatorHub), Kubernetes: RKE2, Redis Version: v7.0.15, Cluster Mode: RedisCluster 3 leaders + 3 followers, API Version: redis.redis.opstreelabs.in/v1beta2
What Happens
After deleting a follower pod, it restarts but stays permanently in slave,fail state. Operator logs show an infinite reconcile loop every ~17 seconds with no resolution — healthy leader count does not match desired, attempting to repair disconnected masters — repeating forever. The operator detects the problem but never fixes it.
Root Cause
When a pod restarts in Kubernetes it gets a new IP. The Redis cluster tracks nodes by IP, so the restarted pod is not recognized. The reconcile logic detects healthy leader count != desired but does not find the failed replica's new pod/IP, verify pod is Ready and Redis is responding, or issue CLUSTER REPLICATE to reattach it to the master.
Current Workaround
Manual fix required after every pod restart by running CLUSTER REPLICATE against the failed follower pod. This is not acceptable for production environments where pod restarts happen regularly due to OOM kills, node drains, or rolling updates.
Ask
Is there a plan to add automatic slave,fail recovery to the RedisCluster controller? The fix in the reconcile loop would be straightforward — if a node is in slave,fail state, find the pod by new IP, check it is Ready and responding to PING, then issue CLUSTER REPLICATE with the master node ID. Other solutions like Bitnami Helm Chart handle this automatically. This is a critical production gap and would greatly improve the reliability of the operator.
All reactions