Valkey Sentinel: get-master-addr-by-name still names the old node after failover. Besides SENTINEL RESET, what do you run?

[efremov-it]

We've been hitting leftover Sentinel advertisements on Valkey in Kubernetes.

ROLE + SET on the data nodes already shows a single writable primary. One Sentinel still answers SENTINEL get-master-addr-by-name with the previous pod, or a dead Pod IP. Clients that land on that Sentinel write into a hole.

The usual hammer is SENTINEL RESET or restart Sentinel. Dual-master is a different problem; we do not send REPLICAOF.

Same leftover pointer as #1322 (the ad never converges). RESET is also the workaround people use on #2158.

We open-sourced a sidecar next to each Sentinel: ROLE+SET as the oracle, heal the pointer through Sentinel's own MONITOR API. Same wire protocol as Redis Sentinel. Default is dry-run.

https://github.com/vpnmesh/redis-sentinel-reconciler
Image: https://hub.docker.com/r/vpnmesh/redis-sentinel-reconciler
Helm: oci://ghcr.io/vpnmesh/charts/redis-sentinel-reconciler

One reconciler per Sentinel.

What do other Valkey operators run for leftover ads besides bouncing Sentinel?

[Kxrma47]

I would not make ROLE/SET an external authority for rewriting Sentinel state: during a partition or promotion window, a node-local role can be transient and bypass Sentinel's epoch/quorum rules.

First compare every Sentinel:

SENTINEL MASTER <master-name>
SENTINEL SENTINELS <master-name>
SENTINEL CKQUORUM <master-name>
INFO sentinel

Check config-epoch, peer visibility, and that all Sentinel-to-Sentinel traffic reaches port 26379. In Kubernetes, give each Sentinel a stable announced address and port; stale Pod IPs usually indicate broken peer discovery or announce configuration. After connectivity is fixed, SENTINEL RESET <master-name> on only the stale Sentinel is the supported way to forget and rediscover that master—persist its writable config so a restart does not restore stale state. Sentinel's state, epoch, and reconfiguration model is documented at https://valkey.io/topics/sentinel/

[efremov-it]

Sentinel picks the master by vote: epoch, quorum, who can see whom on 26379. A node can look like a master for a second during a partition or a failover (SET works) while Sentinel is still arguing. If we told Sentinel "that's your master" in that window, we'd skip its rules. We don't want that.

We don't choose who is writable. That's already a fact on Redis: either SET works on one node, or it doesn't.

We only look at this Sentinel's get-master-addr-by-name and ask: does it name the node that actually takes writes?

If there are two writers, or zero, or a failover is in progress, we do nothing. Default is dry-run anyway.

If the other Sentinels already name the writable node, your path is the right first move: check MASTER / SENTINELS / CKQUORUM, fix announce in Kubernetes, then RESET on the one that's stale. We don't run CKQUORUM ourselves.

The case we actually hit is the other way around. Redis already has one writer, but RESET just learns the same lie back from its peers. Then, and only with apply on, we tell that one Sentinel to MONITOR the node that's already writable. That's fixing what it advertises, not running an election.

If you've seen a promotion window where a node is briefly the only one that accepts SET and apply would MONITOR it, I'd like that repro.

[Kxrma47]

Thanks—that clarification narrows the tool considerably. If it probes every known data node, requires exactly one writable node, and defaults to dry-run, it can be useful for diagnosing a stale Sentinel view.

I would still avoid automatically issuing SENTINEL MONITOR from writability alone. A successful SET is an observer-local fact, not proof that the node won Sentinel's election; during a partition an isolated old primary may remain writable. Prefer non-mutating ROLE/INFO replication checks, require SENTINEL CKQUORUM plus majority Sentinel agreement, and have the tool print the corrective command for operator confirmation. Also describe MONITOR as replacing/reseeding that Sentinel's monitored master, not as completing a failover. Those guardrails make the apply mode much safer while preserving the stale-view repair use case.

[efremov-it]

Yeah. The useful part is the dry-run: hit every known node, want exactly one writer, don't touch Sentinel. That's a stale-view check, not an election.

SET is only "this node took a write from here." An isolated old primary can still do that. That's why REDIS_ADDRS is the whole cluster, and apply bails if too few of those nodes are reachable. A SET that only works on the island you're sitting on is not enough.

CKQUORUM we don't check today. Fair gate if apply is on. If the other Sentinels already name the writable node, RESET on the stale one is the supported fix. The leftover we still hit is when they agree with each other on the wrong node, so RESET just learns it again.

MONITOR on that one process isn't finishing a failover. It's reseeding who that Sentinel monitors. I'll say it that way.

Printing the exact SENTINEL RESET / MONITOR line in dry-run and letting a human paste it is the safer apply story. Default stays look-don't-touch.