perf(node): fix slot-driver starvation under gossip flood (#863)

[ch4r10t33r]

Summary

Issue #863 reported the zeam_0 aggregator on devnet-4 falling badly behind
the rest of the network: head trailed finalized by ~100 slots, the slot
clock missed its tick deadline for seconds at a time, and ~74% of incoming
attestations referenced blocks the node hadn't seen yet — every one of
those attestations triggered a block-fetch RPC that itself timed out, so
the node spent its time chasing blocks instead of running fork choice.

This PR fixes the underlying starvation. It also brings zeam's gossip
attestation handling in line with leanSpec so the behaviour matches the
reference implementation rather than diverging quietly.

What goes wrong today

The aggregator does almost everything on a single thread: receive gossip,
validate attestations against fork choice, run state transitions on new
blocks, fire the every-800ms slot tick, etc. Under load four problems
compound:

1. The event loop drains every queued I/O completion before handing
   control back, which can take seconds when peers are flooding.
2. Each attestation that references an unknown block fans out into a
   point-lookup RPC, four times over for an aggregator subscribed to four
   subnets. The fan-out itself becomes the load.
3. The slot tick is best-effort: if it doesn't get CPU in time, the node
   silently runs fork choice on stale state.
4. Incoming attestations are validated on the same thread that imports
   blocks, so they fight each other for the fork-choice lock.

What this PR changes

The series is layered so each commit is independently reviewable.

1. metrics: add #863 P2/P3/P4 gating + cap counters — registers
   three new metrics consumed by the rest of the series. No behaviour
   change.
2. clock: bound xev drain to one CQE batch per pass (#863 P4) —
   the event loop now returns to the slot driver after one batch of I/O
   completions instead of draining everything in sight. The slot tick is
   still rate-gated to its own cadence, so timing histograms stay
   meaningful.
3. network: cap concurrent BlocksByRoot RPCs in-flight (#863 P3) —
   caps in-flight outbound block-fetch RPCs at 8. Anything beyond that
   is rejected on the spot and counted, so a flood can't run the
   fan-out away.
4. chain: future-slot drop + sync-aware gossip gating (#863 P2) —
   drops two classes of attestation cheaply on the receive path: ones
   for slots that haven't happened yet, and ones that arrive while the
   node is still catching up to peers. Block gossip is intentionally not
   gated — that's the recovery vector during sync.
5. chain: move gossip attestation validation to chain-worker (#863 P5) — the receive path becomes pure dispatch. The expensive
   work (fork-choice reads, signature verification) moves onto a
   dedicated chain-worker thread so it stops competing with block
   import.
6. chain: align gossip attestation lifecycle with leanSpec (#863 spec parity) — closes the gaps between zeam and the
   reference implementation:
   • Attestations whose referenced block isn't yet imported, or whose
     slot is just slightly in the future, are now buffered (capped at
     1024 entries, FIFO eviction) and replayed after every successful
     block import — same as leanSpec's _pending_attestations /
     _replay_pending_attestations.
   • Sync-state gating now matches the spec: while the node is syncing
     it accepts gossip and uses the buffer to absorb out-of-order
     arrivals; only IDLE-equivalent states (no peers, fork choice not
     initialised) drop gossip.
   • Non-aggregators now run validation and signature verification on
     received attestations and then drop them, matching the spec's
     "validate and drop" path. Storage in fork choice stays gated on
     the aggregator role.
7. build: invoke rust nightly via 'rustup run' … + build: fix zig fmt of rustup run cargo argv — CI fix for macos-latest
   where cargo +nightly was hitting the rustup installer instead of
   the proxy and failing immediately. Switching to
   rustup run nightly cargo … works on every supported install
   without depending on which binary cargo happens to point at.

What this PR does not change

A few things look related but are deliberately out of scope:

• The fork-choice attestation tracker (zeam's per-validator latestNew
  / latestKnown) stays as the head-selection mechanism. The spec
  doesn't define it and zeam relies on it; this PR doesn't touch that
  shape.
• The BlocksByRoot in-flight cap and the one-batch event-loop bound
  are zeam-specific performance knobs the spec is silent on; they stay.
• The --aggregate-subnet-ids selection policy (own + neighbour) is a
  separate change that lives in lean-quickstart, not here.

New metrics

• zeam_gossip_atts_dropped_total{kind, reason} — drops on the receive
  thread before chain-worker dispatch (syncing / worker_validation_failed).
• zeam_blocks_by_root_inflight — instantaneous outbound block-fetch
  RPC count, capped at 8.
• zeam_xev_clock_drain_passes_total — passes through the bounded
  event-loop drain.
• lean_pending_attestations_buffered_total{kind, reason} — entries
  pushed into the spec-style replay buffer.
• lean_pending_attestations_evicted_total{kind} — FIFO evictions at
  the 1024 cap.
• lean_pending_attestations_replay_total{kind, outcome} — replay
  outcomes (accepted / buffered / dropped).
• lean_pending_attestations_size{kind} — instantaneous buffer depth.

lean_block_fetch_dedup_total gains a new outcome="inflight_cap"
label so the existing "every dedup outcome is accounted for" invariant
still holds.

Test plan

• zig build clean on macOS aarch64.
• zig build test — all test binaries pass (271 tests across 14
  binaries).
• zig fmt --check . clean.
• CI on perf/863-loop-fairness-and-fetch-coalesce after the
  rustup-invocation fix: lint and Dummy prove (macos-latest)
  passing on the latest commit; build / test /
  build-all-provers running.
• Devnet redeploy — watch for:
  * zeam_blocks_by_root_inflight bounded by 8 under load.
  * lean_block_fetch_dedup_total{outcome="inflight_cap"} only
  non-zero during flood, zero in steady state.
  * zeam_gossip_atts_dropped_total{reason="syncing"} clears as
  nodes catch up.
  * lean_pending_attestations_size non-zero only during sync /
  during reorgs; replay outcomes weighted toward accepted.
  * Slot-tick latency back below the slot duration on the
  aggregator.
  * Head-vs-finalized gap stays bounded (vs ~100 slots pre-fix).

Refs #863

[zclawz]

Adversarial review of #886:

Finding

High — aggregated gossip clone leaks when worker queue rejects submission

In the new aggregation worker-dispatch path, signed_aggregation is deep-cloned before submitGossipAggregatedAttestation. The cleanup guard is currently an errdefer:

var cloned: types.SignedAggregatedAttestation = undefined;
try types.sszClone(self.allocator, types.SignedAggregatedAttestation, signed_aggregation, &cloned);
var cloned_consumed = false;
errdefer if (!cloned_consumed) cloned.deinit();
self.submitGossipAggregatedAttestation(cloned) catch |err| switch (err) {
    error.QueueFull => { ... return .{}; },
    error.QueueClosed => { ... return .{}; },
    error.ChainWorkerDisabled => unreachable,
};
cloned_consumed = true;

QueueFull and QueueClosed return .{} from onGossip (a successful return), so the errdefer does not run. Because cloned_consumed remains false and the worker never takes ownership, the clone's owned proof.participants / signature allocations leak for every rejected aggregated-attestation gossip message.

That is especially dangerous for this PR because the queue-backpressure path is intentionally expected under the #863 gossip-flood scenario. Under sustained aggregation flood, the fix can convert bounded queue drops into unbounded heap growth on the libxev ingress path.

Suggested fix: make the guard a normal defer scoped around the dispatch attempt, or explicitly cloned.deinit() before the return .{} branches. For example:

var cloned_consumed = false;
defer if (!cloned_consumed) cloned.deinit();
...
cloned_consumed = true;

(or set cloned_consumed = true only after a successful enqueue).

Validation

• I inspected the full PR diff, focusing on pkgs/node/src/chain.zig, pkgs/node/src/network.zig, pkgs/node/src/clock.zig, and metrics additions.
• Started zig build test; it reached the long node stress harness successfully, but I am not waiting on the full suite here because this ownership bug is already merge-blocking.

[zclawz]

GrapeBaBa review verification ✅ (HEAD cd245a5)

1. CI yml (cache: false / cache-bin: false) — context only
Addressed via explanation. The changes are a macOS runner prerequisite unrelated to the #863 performance work. No code change needed.

---

2. clock.zig:215 — early tickInterval() skipping slot duties ✅ Fixed

• Issue: outer check used + CLOCK_DISPARITY_MS slack, so .once could call tickInterval() during the slack window while the interval timer was still pending → timer canceled → onInterval() skipped.
• Fix: outer check now uses a strict boundary — current_interval_time_ms + SECONDS_PER_INTERVAL_MS <= time_now_ms (no slack). CLOCK_DISPARITY_MS slack is preserved only inside tickInterval's while-loop for multi-slot catch-up. Lines 70–80 in the diff explicitly document this as the perf(node): fix slot-driver starvation under gossip flood (#863) #886 review catch.

---

3. chain.zig:808 — replayPendingAttestations() only on chain-worker path ✅ Fixed

• Issue: replay only triggered from chainWorkerOnBlockThunk; direct import paths (blocks_by_root/blocks_by_range, pending-block queue, published blocks, --chain-worker=false) never called it.
• Fix: replayPendingAttestations() now called at 3 call sites:
  • chainWorkerOnBlockThunk (~line 811, original) ✅
  • processPendingBlocks success path (~line 1885) ✅
  • Non-worker direct onBlock path — gossip + published blocks (~line 2504) ✅

All three concerns addressed. PR looks good to merge from a review-fix standpoint.