meta: detect a datanode that restarted in place - #64
Merged
Conversation
server_heartbeat overwrote server.boot_time_ms with whatever the latest heartbeat claimed and never compared it to anything, so a datanode that crashed and came back between two heartbeats was completely invisible to the metaserver. That is a routing correctness hole, not just a missing metric. The restarted process has dropped every shard the metaserver still believes it is serving, so reads routed there miss until it finishes reloading - and nothing in the stale-timeout path will ever notice, because the heartbeats never stopped. Only silence is detectable today, and a restart is precisely the failure that produces no silence. The metaserver now anchors on the first non-zero boot time a registered server reports, and flags a later heartbeat that claims a different one as a reboot. ServerMetaInfo carries the anchor (reported_boot_time_ms) alongside the latest claim (boot_time_ms); the two disagreeing is the signal. The anchor deliberately does not follow the new value, which is what makes the verdict sticky rather than resetting itself on the next beat. It clears on re-registration, which is how a datanode announces it is ready to be trusted again. Both fields carry serde defaults, so existing snapshots load unchanged. A boot time of zero never flags anything. Older datanodes report zero, and treating that as a changed boot time would convict the whole fleet during an upgrade. The conviction planner treats a reboot as a failure (ConvictionPolicy:: convict_on_reboot, on by default, TS_META_CONVICT_ON_REBOOT to disable). Two properties matter here: Reboot conviction is not gated on the detector being active. The stall guard exists because silence is ambiguous after the detector itself pauses; a changed boot time is direct evidence that the process restarted, so the guard does not apply and a restart is actionable on the very first round. Reboot conviction is still gated by safe mode. A rolling restart that takes out half a rack is as damaging as a rack fault and has to hit the same brake. ConvictionPlan reports the rebooted servers separately from the convicted ones, so a restart stays visible whether or not the policy acts on it - including when convict_on_reboot is off. Detection itself is always on and only adds two fields plus a topology event; nothing acts on the flag unless the adaptive failure detector is enabled, which is still off by default. 7 new tests: heartbeat anchoring, stickiness across repeats, the clear on re-registration, zero boot times never flagging, a restart being convicted with no silence at all, a restart still being held by safe mode, reboot conviction being switchable off while still reported, a restart surviving the detector pause, and an already-frozen server not being reconvicted for a reboot. Based on the adaptive failure detector branch, since it wires into the conviction planner introduced there.
This was referenced Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
server_heartbeatoverwroteserver.boot_time_mswith whatever the latest heartbeat claimed and never compared it to anything, so a datanode that crashed and came back between two heartbeats was completely invisible to the metaserver.That is a routing correctness hole, not a missing metric. The restarted process has dropped every shard the metaserver still believes it is serving, so reads routed there miss until it finishes reloading — and nothing in the stale-timeout path will ever notice, because the heartbeats never stopped. Only silence is detectable today, and a restart is precisely the failure that produces no silence.
What this adds
The metaserver anchors on the first non-zero boot time a registered server reports, and flags a later heartbeat claiming a different one as a reboot.
ServerMetaInfocarries the anchor (reported_boot_time_ms) alongside the latest claim (boot_time_ms); the two disagreeing is the signal. The anchor deliberately does not follow the new value — that is what makes the verdict sticky rather than resetting itself on the next beat. It clears on re-registration, which is how a datanode announces it is ready to be trusted again. Both fields carry#[serde(default)], so existing snapshots load unchanged.A boot time of zero never flags anything. Older datanodes report zero, and treating that as a changed boot time would convict the whole fleet during an upgrade.
How it feeds conviction
The planner treats a reboot as a failure (
ConvictionPolicy::convict_on_reboot, on by default). Two properties matter:ConvictionPlanreports rebooted servers separately from convicted ones, so a restart stays visible whether or not the policy acts on it — including whenconvict_on_rebootis off.Detection itself is always on: it adds two fields and a topology event, and nothing acts on the flag unless the adaptive failure detector is enabled, which is still off by default.
TS_META_CONVICT_ON_REBOOT1Tests
7 new tests: heartbeat anchoring, stickiness across repeats, the clear on re-registration, zero boot times never flagging, a restart being convicted with no silence at all, a restart still being held by safe mode, reboot conviction being switchable off while still reported, a restart surviving the detector pause, and an already-frozen server not being reconvicted for a reboot.
Verification:
cargo test -p temporalstore-rust --lib meta— 110 passed, 0 failed. (The base branch scores 101 passed + 2 flakyproxy::testsfailures that also fail on pristinemain; both passed this run.)cargo test -p temporalstore-rust --bin metaserver— 18 passed, 0 failed.cargo build -p temporalstore-rust --bin metaserver— clean, no new warnings.