fix(spurctld): stop serving when the raft core is dead - #810
Conversation
A panic inside RaftCore ended that task alone. Every other task kept running, so the process stayed up, the gRPC listener kept accepting connections, and the replica served stale reads while every write failed. Kubernetes could not see it either: the probes were tcpSocket on 6817, which the listener answers whatever Raft is doing. Observed as a Pod that stayed 1/1 Running with 0 restarts while it replicated nothing. Two changes, because one alone is not enough. openraft moves the metrics sender into RaftCore and leaves only the receiver on the handle, so the channel closing is an exact signal that the task has ended. spurctld now watches it and exits 70, and the supervisor restarts a whole process. The metrics server also serves /healthz, which reports whether the core runs, and /readyz, which adds a known leader so a replica that cannot answer a read leaves the Service. The example manifests probe these instead of the TCP port, and set metrics bind = "all" so the kubelet can reach them. The headless Service gains publishNotReadyAddresses: true. Peer DNS has to resolve before a pod is ready, because the replicas need each other to elect the leader that readiness waits for; without it the two conditions deadlock at bootstrap and no cluster ever forms.
The headless spurctld Service sets publishNotReadyAddresses, because the replicas need peer DNS before they can elect a leader. That flag also publishes a replica whose Raft is dead, so the readiness probe changed nothing for a client. Add spurctld-client, a normal Service without the flag. It drops a replica while /readyz fails. The operator now uses it.
/healthz and /readyz lived on the metrics server. metrics.bind defaults to loopback and metrics.enabled can turn the server off, so a kubelet probe against that port reaches nothing and every replica stays unready for good. Behind a Service that honours readiness this is a full outage, caused by a default. Add [health], enabled and bound to all interfaces by default, on port 6823. The two routes answer with a status code and one word, so they carry nothing worth hiding. They stay mounted on the metrics port too, for a local check. The example Pod probes now use 6823, and the example config no longer has to publish metrics to the Pod network to make its probes work.
|
Verified on a cluster: The description says the Method. Three node RKE2 cluster, in a namespace of its own so the healthy deployment stayed untouched.
Result on Container Before this change the same panic left the Pod One note for anyone reproducing it: the panic needs a genuinely stale store. A first attempt without the |
1 participant
Related:
Motivation
A
spurctldwhoseRaftCorehas stopped keeps running and keeps taking work.RaftCoreis a task of its own. When it panics, that task ends and every othertask survives, the gRPC listener included. The process therefore goes on accepting
connections and serving reads from a state machine nothing can replicate to, while
every write fails. The readiness probe is
tcpSocket: 6817, which reaches thelistener and not Raft, so Kubernetes sees a healthy Pod:
1/1 Running, 0 restarts,and the Service keeps sending clients to it.
The goal is that a controller which cannot replicate stops taking traffic, and then
stops.
Technical Details
Leaving when the core dies.
openraftowns the metrics watch channel frominside
RaftCore, so the channel closes exactly when that task ends, whateverended it.
RaftHandle::core_stoppedawaits that close; a supervisor task inmain.rslogs the reason and exits with code 70, so a supervisor restarts a wholecontroller instead of leaving a half dead one in service.
A probe that reads Raft.
/healthzreports whether the core runs./readyzadds a known leader, because a replica that knows no leader cannot answer a read
that means anything.
A listener of its own for health. The first version of this change put the two
routes on the metrics HTTP server. That was wrong:
metrics.binddefaults toloopbackandmetrics.enabledcan turn that server off, so with the defaults akubelet probe reaches nothing and every replica stays unready for good. A new
[health]section, enabled by default and bound to every interface on port 6823,carries them instead. The two routes answer with a status code and one word, so
they carry nothing worth hiding, unlike the metrics they used to share a port with.
They stay mounted on the metrics port as well, for a local check.
Note for reviewers: this opens a new listener on an existing deployment after
upgrade. It is documented in
docs/deployment/upgrading.rstanddocs/deployment/native-host.rst, andhealth.enabled = falseswitches it off.If you would rather it defaulted to loopback, say so, but be aware that then a
deployer who forgets to override it gets every replica permanently unready, which
is the failure this section exists to prevent.
Two Services. The replicas need peer DNS before a leader exists, and
readiness waits for a leader, so the headless Service must set
publishNotReadyAddressesor the two conditions deadlock and the cluster neverstarts. That flag also publishes an unready replica, which would make the readiness
probe pointless for clients. A second Service,
spurctld-client, has no such flagand therefore drops a replica while
/readyzfails. The operator uses it.Related:
Test Plan
deadlock.
service, and that the old TCP probe would have passed it.
and that
spurctld-clientdoes not.[health]section, to prove the default workswithout the deployer knowing about it, and that metrics stay on loopback.
spurctld-client, and confirm accounting records it.readiness gate causes an outage of its own during an election.
Test Result
Environment: three node RKE2 cluster on cloud VMs, 8 vCPU and 96 GiB each, no GPU,
with PostgreSQL accounting.
1/1 Runningon all three in 40 s.0/1 Running. Itscontainer was
Started, so the oldtcpSocket: 6817probe would have passed it,while the kubelet logged
Readiness probe failed: HTTP probe failed with statuscode: 50325 times.headless Service still published the address for peer DNS
(
ready:true, serving:false), andspurctld-clientpublished nothing, so noclient could reach it.
carrying no
[health]section. The log readsmetrics ... bound=127.0.0.1:6822beside
health ... bound=[::]:6823, and all three replicas were Ready. Metricsstayed on loopback and the probes still answered, which is the whole point of the
separate listener.
spurctld-clientcompleted, andsacctshows it.endpoint count of
spurctld-clientwas sampled every 0.45 s. It went 3, then 2,then back to 3 after 43 s. It never reached 0, so the readiness gate costs nothing
during an election. A job submitted afterwards completed on the new leader.
cargo clippy --workspace --exclude spur-ffi --all-targets --lockedreportsnothing;
cargo test --lockedpasses 3525 tests, including new ones for the twoendpoints and for the health defaults.
sphinx-build -W --keep-goingsucceeds.Not verified: the
exit(70)path has never been seen to fire on a cluster.RaftCorepanicked reliably only through the scale-up defect, and the change inthe related bootstrap PR removes that route, so the condition can no longer be
produced on a real cluster. It is covered by unit tests only, and this is stated
rather than glossed over.
Submission Checklist