Skip to content

meta: detect a datanode that restarted in place - #64

Merged
matrixarkai merged 1 commit into
mainfrom
oss/meta-reboot-detection
Aug 19, 2026
Merged

meta: detect a datanode that restarted in place#64
matrixarkai merged 1 commit into
mainfrom
oss/meta-reboot-detection

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Builds on #61 (merged): it wires into the conviction planner introduced there.

The problem

server_heartbeat overwrote server.boot_time_ms with whatever the latest heartbeat claimed and never compared it to anything, so a datanode that crashed and came back between two heartbeats was completely invisible to the metaserver.

That is a routing correctness hole, not a missing metric. The restarted process has dropped every shard the metaserver still believes it is serving, so reads routed there miss until it finishes reloading — and nothing in the stale-timeout path will ever notice, because the heartbeats never stopped. Only silence is detectable today, and a restart is precisely the failure that produces no silence.

What this adds

The metaserver anchors on the first non-zero boot time a registered server reports, and flags a later heartbeat claiming a different one as a reboot.

ServerMetaInfo carries the anchor (reported_boot_time_ms) alongside the latest claim (boot_time_ms); the two disagreeing is the signal. The anchor deliberately does not follow the new value — that is what makes the verdict sticky rather than resetting itself on the next beat. It clears on re-registration, which is how a datanode announces it is ready to be trusted again. Both fields carry #[serde(default)], so existing snapshots load unchanged.

A boot time of zero never flags anything. Older datanodes report zero, and treating that as a changed boot time would convict the whole fleet during an upgrade.

How it feeds conviction

The planner treats a reboot as a failure (ConvictionPolicy::convict_on_reboot, on by default). Two properties matter:

  • Not gated on the detector being active. The stall guard from meta: adaptive failure detection with a correlated-failure guard #61 exists because silence is ambiguous after the detector itself pauses. A changed boot time is direct evidence that the process restarted, not an inference drawn from silence, so the guard does not apply and a restart is actionable on the very first round.
  • Still gated by safe mode. A rolling restart that takes out half a rack is as damaging as a rack fault and has to hit the same brake.

ConvictionPlan reports rebooted servers separately from convicted ones, so a restart stays visible whether or not the policy acts on it — including when convict_on_reboot is off.

Detection itself is always on: it adds two fields and a topology event, and nothing acts on the flag unless the adaptive failure detector is enabled, which is still off by default.

Variable Default Meaning
TS_META_CONVICT_ON_REBOOT 1 Treat a detected reboot as a failure

Tests

7 new tests: heartbeat anchoring, stickiness across repeats, the clear on re-registration, zero boot times never flagging, a restart being convicted with no silence at all, a restart still being held by safe mode, reboot conviction being switchable off while still reported, a restart surviving the detector pause, and an already-frozen server not being reconvicted for a reboot.

Verification:

  • cargo test -p temporalstore-rust --lib meta110 passed, 0 failed. (The base branch scores 101 passed + 2 flaky proxy::tests failures that also fail on pristine main; both passed this run.)
  • cargo test -p temporalstore-rust --bin metaserver — 18 passed, 0 failed.
  • cargo build -p temporalstore-rust --bin metaserver — clean, no new warnings.

server_heartbeat overwrote server.boot_time_ms with whatever the latest
heartbeat claimed and never compared it to anything, so a datanode that crashed
and came back between two heartbeats was completely invisible to the
metaserver.

That is a routing correctness hole, not just a missing metric. The restarted
process has dropped every shard the metaserver still believes it is serving, so
reads routed there miss until it finishes reloading - and nothing in the
stale-timeout path will ever notice, because the heartbeats never stopped. Only
silence is detectable today, and a restart is precisely the failure that
produces no silence.

The metaserver now anchors on the first non-zero boot time a registered server
reports, and flags a later heartbeat that claims a different one as a reboot.
ServerMetaInfo carries the anchor (reported_boot_time_ms) alongside the latest
claim (boot_time_ms); the two disagreeing is the signal. The anchor deliberately
does not follow the new value, which is what makes the verdict sticky rather
than resetting itself on the next beat. It clears on re-registration, which is
how a datanode announces it is ready to be trusted again. Both fields carry
serde defaults, so existing snapshots load unchanged.

A boot time of zero never flags anything. Older datanodes report zero, and
treating that as a changed boot time would convict the whole fleet during an
upgrade.

The conviction planner treats a reboot as a failure (ConvictionPolicy::
convict_on_reboot, on by default, TS_META_CONVICT_ON_REBOOT to disable). Two
properties matter here:

Reboot conviction is not gated on the detector being active. The stall guard
exists because silence is ambiguous after the detector itself pauses; a changed
boot time is direct evidence that the process restarted, so the guard does not
apply and a restart is actionable on the very first round.

Reboot conviction is still gated by safe mode. A rolling restart that takes out
half a rack is as damaging as a rack fault and has to hit the same brake.
ConvictionPlan reports the rebooted servers separately from the convicted ones,
so a restart stays visible whether or not the policy acts on it - including when
convict_on_reboot is off.

Detection itself is always on and only adds two fields plus a topology event;
nothing acts on the flag unless the adaptive failure detector is enabled, which
is still off by default.

7 new tests: heartbeat anchoring, stickiness across repeats, the clear on
re-registration, zero boot times never flagging, a restart being convicted with
no silence at all, a restart still being held by safe mode, reboot conviction
being switchable off while still reported, a restart surviving the detector
pause, and an already-frozen server not being reconvicted for a reboot.

Based on the adaptive failure detector branch, since it wires into the
conviction planner introduced there.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants