Duty sync gate reports Synced while the node is far behind, so it attests and proposes on a stale head

[MegaRedHand]

Summary

A node that is far behind the chain reports Synced, not Syncing, so duties_allowed() returns true and it attests and proposes on a head that can be hundreds of slots old. The misread persists for the entire catch-up, not just a tick or two.

Why it happens

BlockChainServer::update_sync_status (crates/blockchain/src/lib.rs:1260) derives the "freshest block the network knows about" from the node's own imported chain:

let max_seen_slot = self
    .store
    .max_live_chain_slot()
    .expect("max live chain slot exists")
    .unwrap_or(head_slot);

max_live_chain_slot scans the LiveChain table, and insert_pending_block (crates/storage/src/store.rs:1134) deliberately skips LiveChain, so gossip blocks whose parents are still missing never raise it. While the node backfills, max_seen_slot rises only in lockstep with head_slot.

SyncStatusTracker::update (crates/blockchain/src/sync_status.rs:98) then hits the network-stall escape hatch:

if network_lag > NETWORK_STALL_THRESHOLD {   // 8
    self.syncing = false;                    // "the network stalled, keep validating"
}

With a gap of, say, 550 slots, network_lag = 550 > 8, so the node declares itself Synced. The escape hatch exists so validators keep working through a genuine network-wide stall, but it cannot distinguish "the network is stalled" from "I am far behind and have imported nothing recent" — both look like a large current_slot - max_seen_slot.

sync_status_treats_stale_known_blocks_as_network_stall pins exactly this: update(100, 0, 0) == Synced.

Why now

This is pre-existing, not a regression: before #554 a node restarted without --checkpoint-sync-url clobbered its DB with a genesis anchor and hit the same misread from slot 0.

What changed is that #554 makes resume-from-a-stale-DB a supported recovery path and documents it as the safe option when no checkpoint URL is available (docs/checkpoint_sync.md, "Restarts and Existing State"). So the sharp edge is now something operators are steered into rather than something that only happened when they had already lost their chain.

Impact

For the duration of a backfill, the node:

• produces attestations for a head hundreds of slots behind the canonical one
• proposes blocks on that old head if it holds a proposer slot

Both are wasted at best, and add fork-choice noise the rest of the network has to absorb.

Possible directions

Not a recommendation, just the shape of the options:

1. Track peers' advertised heads (Status req-resp already carries them) and use that as max_seen_slot, instead of the node's own imported chain. Makes "the network is ahead of me" distinguishable from "the network is stalled".
2. Let pending blocks raise the freshest-known slot without entering LiveChain, e.g. a separate high-water mark updated in insert_pending_block.
3. Bound the network-stall escape hatch: above some much larger lag, treat it as local rather than network-wide, since a real network-wide stall does not usually run for hundreds of slots while gossip keeps arriving.

Workaround

Pass --checkpoint-sync-url when a node is known to be far behind. It skips the backfill entirely, so the gate never misreads.

References

• fix(startup): resume from existing DB without a checkpoint URL #554 (resume from existing DB without a checkpoint URL) — documents the caveat
• crates/blockchain/src/sync_status.rs
• crates/blockchain/src/lib.rs:1260 (update_sync_status)
• crates/storage/src/store.rs:1134 (insert_pending_block), :984 (max_live_chain_slot)