node: harden blocks_by_range catch-up retry and fork recovery

[ch4r10t33r]

Summary

Fixes #893. Hardens blocks_by_range bulk catch-up and the chain-worker backpressure path so a fresh-genesis or aggregator node cannot get stuck with a frozen head while the slot clock keeps advancing. Three follow-up commits were folded into this PR after the initial review to address regressions found on devnet.

blocks_by_range catch-up (#893)

• Fork mismatch: abort range batch when the first chunk at start_slot does not parent-link to our head; fall back to blocks_by_root from peer head.
• Retry/fallback: on range failure, timeout, or zero imports — retry on another range-capable peer (up to 3 attempts), then blocks_by_root from peer head.
• blocks_by_range unavailable: if a peer returns unsupported / invalid-request (or dispatch fails), mark the peer and fall back to blocks_by_root immediately instead of retrying range on that peer (per Anshal).
• Gap sanity: cap sync gap with min(peer_head - our_head, wall_slot - our_head).
• Early devnet: trigger catch-up on head gap even when both sides report finalized_slot == 0.
• Pagination: chain another range after a successful batch if still far behind peer head.
• Concurrency: one in-flight blocks_by_range per (start_slot, count) window; disjoint ranges may run in parallel.
• Async import accounting: count chunks_imported only after chain-worker onBlock succeeds; defer sync-end until async imports drain.
• Helpers: pure gap/retry logic in blocks_by_range_sync.zig (orchestration remains in `node.zig` for now).

Aggregated-attestation queue backpressure

Under devnet restart load the chain-worker aggregated-attestation queue (512) could fill while block STF and catch-up saturated the single worker thread, causing permanent gossip drops (`aggregated attestation queue full, dropping slot=…`) and starving fork-choice vote tracking.

• Buffer on `QueueFull`: mirror leanSpec pending-buffer semantics — enqueue to `pending_aggregated_attestations` with reason `queue_full` and nudge `replay_pending_attestations` instead of dropping.
• Batch drain: when agg queue depth exceeds 16, drain up to 32 aggregations per worker loop iteration (still one block + one raw attestation per iteration). Does not call `forkChoice.aggregate()`; committee aggregation still uses the `AggregateSnapshot` snapshot path (Investigate slow slot_interval / tick duration (event-loop starvation vs nominal 0.8s) #863/node, metrics: offload heavy chain mutations to chain-worker, parallelize XMSS verify (#863) #890).
• Per-agg work: decode participant bits once; pass indices into verify; drop redundant `validator_ids` allocation.

Post-#894 follow-ups (regressions found on devnet)

After the original PR landed, two regression patterns were observed on devnet (most visibly on `zeam_4`/`zeam_8` aggregators) and addressed in three follow-up commits:

1. Sync gated on host wall clock + slot-driver watchdog (`77d90292`)

Symptom: `shouldCatchUpFromPeerStatus` was reading `forkchoice.fcStore.slot_clock.timeSlots` to decide whether to catch up, but on aggregators libxev could stall and freeze that clock; once frozen, the node could no longer detect that it was behind and never re-armed catch-up.

• New `Clock.wallSlotNow()` derives the current slot from host `std.time` + genesis time, decoupled from the libxev-driven slot clock.
• `shouldCatchUpFromPeerStatus` now uses `wallSlotNow()` so that wall-clock gap drives catch-up even when the slot driver is hung.
• Added `SlotDriverWatchdog` (background `std.Thread`): observes the libxev slot driver's last-tick timestamp via an atomic, fires when it exceeds the threshold (default 5 s, see Investigate slow slot_interval / tick duration (event-loop starvation vs nominal 0.8s) #863), and emits `zeam_slot_driver_stall_fired_total` / `zeam_slot_driver_stall_seconds`. When fired, a registered callback in `BeamNode` forces `refreshSyncFromPeers()` so a stalled main loop can recover instead of silently lagging.

2. Reverted `getSyncStatus` head-gap check (`6d8a3920`)

A short-lived attempt to mark a node `.behind_peers` when `our_head_slot < max_peer_head_slot` caused validators to skip their duties when other clients on the network briefly ran ahead. Reverted; sync status now keeps the original spec-aligned semantics and only signals `.behind_peers` for finality / sync-distance signals, not transient head jitter.

3. Drop RPC catch-up chunks on chain-worker `QueueFull` instead of inline-importing (`b2654679`)

This is the regression that PR #894 itself introduced and that the user reported as "all zeam nodes head slot are far behind … clear regression". Diagnosed from `zeam_8` logs as a 9.7-second libxev event-loop stall: a 3.4 MB `blocks_by_root` response was being processed inline on the libxev thread because the chain-worker block queue was already saturated by attestation traffic on the aggregator subnet, and the RPC chunk handler had no backpressure path — it just fell through to a synchronous `chain.onBlock` (~0.5 s of XMSS verification per block). The gossip path already drops on `QueueFull`; the RPC path was asymmetric.

Fix:

• New `ImportSubmitOutcome` enum (`submitted` / `queue_full` / `worker_disabled` / `failed`) returned by `trySubmitImportToWorker` in `pkgs/node/src/node.zig`.
• New `ChunkImportDisposition` enum + `classifyChunkImport` helper in `pkgs/node/src/blocks_by_range_sync.zig`.
• `processBlockByRootChunk` and `processBlockByRangeChunk` now consult `classifyChunkImport`:
  • `submitted` → `handled` (track range-async import as before)
  • `queue_full` → `drop_backpressure` — log + bump `lean_chain_queue_dropped_total{queue="block"}` (already done by `chain_worker.sendBlock`) and return without inline-importing. For range chunks, deliberately do not advance the by-range request progress, so the chunk-timeout / status-driven retry refetches once the worker queue drains.
  • `worker_disabled` / `failed` → `fallback_inline` (only path that still runs `chain.onBlock` on the caller, as before).
• Aggregators are now bounded by chain-worker capacity instead of pinning the libxev thread when the worker is full.
• Regression tests in `blocks_by_range_sync.zig` assert that every `ImportSubmitOutcome` variant has a defined disposition and that `queue_full` always maps to `drop_backpressure` (never `fallback_inline`).

Test plan

• `zig fmt --check .`
• `cargo fmt --check` / `cargo clippy --workspace -- -D warnings` (rust/Cargo.toml)
• `zig build test --summary all`
• `zig build simtest --summary all`
• Unit test: `cappedSyncGapSlots limits range catch-up to wall-clock head`
• Unit test: `classifyChunkImport: queue_full drops, never falls back to inline (node: harden blocks_by_range catch-up retry and fork recovery #894 regression guard)`
• Unit test: `classifyChunkImport` exhaustive switch over every `ImportSubmitOutcome` variant
• Local-devnet smoke (5 distinct clients × 4 subnets, `0xpartha/zeam:local` built from `b2654679`):
  • 0 `chain-worker block queue full for RPC chunk` warnings
  • 0 `sszClone … falling back to inline import` warnings
  • `lean_chain_queue_dropped_total` never emitted (zero increments)
  • `lean_pending_attestations_size{kind="attestation"} = 0` throughout
  • `verify_signatures` p99 < 1 s (all imports went through chain-worker, none inline on libxev)
  • Local mesh did not stay up long enough on macOS Docker QUIC to drive finality (known `xev` / Docker limitation; see Dockerfile note + Investigate slow slot_interval / tick duration (event-loop starvation vs nominal 0.8s) #863) — the Linux devnet remains the canonical environment for the regression test below.
• Deploy on ansible devnet `zeam_0` (fresh genesis, no checkpoint) and confirm head advances without manual restart
• Confirm `lean_chain_queue_dropped_total{queue="aggregated_attestation"}` stays near zero and `lean_pending_attestations_buffered_total{reason="queue_full"}` is bounded under gossip burst
• Deploy `b2654679` on aggregator `zeam_8` and confirm:
  • no recurrence of multi-second libxev stalls coinciding with large `blocks_by_root` responses
  • `zeam_slot_driver_stall_fired_total` does not increase
  • `lean_chain_queue_dropped_total{queue="block"}` may briefly increment under sustained load — that is the intended backpressure path, not a regression

[zclawz]

Reviewed PR #894. I’m not approving yet — I found two catch-up correctness issues that look realistic enough to block this until fixed.

Blocking findings

1. Overlapping blocks_by_range windows can strand sync completion

   blocks_by_range_active is keyed by exactly (start_slot, count) (pkgs/node/src/network.zig:769-786), so two requests with the same start_slot but different count — or otherwise overlapping slot windows — can run concurrently.

   Async import tracking is block_root -> request_id (pkgs/node/src/node.zig:276-280) and increments chunks_async_pending before inserting into that map. If two overlapping ranges submit the same block to the worker before import completes, the later put overwrites the earlier request id. The earlier request’s async-pending count is then never decremented, so sync_end_pending && chunks_async_pending == 0 never becomes true and its finalize/retry/fallback path can hang indefinitely.

   Suggested fixes: either reject overlapping range windows, track block_root -> list/set(request_id), or only increment async-pending when the root was newly tracked for that request.

2. Fork-mismatch detection misses skipped-slot first chunks

   The range fork guard only checks parent linkage when signed_block.block.slot == view.start_slot (pkgs/node/src/node.zig:1573-1584). But blocks_by_range responses legitimately skip empty slots. If the first returned block is at start_slot + N, a sibling-fork first chunk bypasses this guard and falls into missing-parent/cache/fetch handling instead of immediately aborting the range and falling back to blocks_by_root.

   Suggested fix: track whether a chunk is the first received chunk for the request and apply the parent_root == our_head_root_at_start check to that first chunk regardless of its slot.

Checks

• zig fmt --check . ✅
• zig build test --summary all could not complete in this environment because the build invokes rustup, which is not installed here:
  error: failed to spawn and capture stdio from rustup: FileNotFound
  Some non-Rust test targets did run and passed before that failure, but this is not a full test pass.

Once the two range-sync issues are addressed, I’m happy to re-review quickly.

[zclawz]

Re-reviewed at 1620e637f53180cf8e630abc2e10d9a9cee51f91 after the follow-up fix. The two blockers from my earlier review are addressed:

• overlapping range windows are now rejected via overlap-aware active range tracking (rangesOverlap + reserveBlocksByRangeActive), and duplicate async chunk tracking no longer increments pending accounting for an already-tracked root;
• the fork-mismatch guard now applies to the first returned chunk regardless of skipped slots via is_first_chunk.

I don’t see remaining blockers from my review pass. Approved from my side / LGTM.

Validation I could run here:

• zig fmt --check . ✅
• zig build test --summary all still cannot fully complete in this environment because rustup is not installed (failed to spawn ... rustup: FileNotFound). The non-Rust/cached tests that ran passed, but this is not a substitute for CI/devnet validation.