Skip to content

Add ws cap liveness check#296

Merged
KirillPamPam merged 5 commits into
mainfrom
head-liveness
Jul 23, 2026
Merged

Add ws cap liveness check#296
KirillPamPam merged 5 commits into
mainfrom
head-liveness

Conversation

@KirillPamPam

@KirillPamPam KirillPamPam commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Head-liveness gating of the WebSocket subscription capability (WsCap)

Summary

An EVM upstream whose head is driven by a WebSocket must now prove its head is live
advancing consecutively — before it advertises the base WebSocket capability WsCap and can
back client subscriptions. A head that skips blocks or stalls, or a WebSocket disconnect, pulls
the upstream out of subscription serving until it recovers; its regular RPC routing is
untouched. The behavior is controlled by a new option,
disable-liveness-subscription-validation — its default is mode-dependent: validation is
off in default mode and on in strict mode.

Motivation

Ported conceptually from dshackle's HeadLivenessValidator. Until now WsCap was asserted as
soon as the WebSocket connector reported connected, so an upstream whose newHeads
subscription was wedged or delivering gappy/stale heads could still be selected as a
subscription source and relay a bad stream to clients. This adds a liveness signal specific to
ws-driven heads.

Like dshackle, head liveness is deliberately not an availability/health signal: it feeds
only the subscription capability, never normal request routing. WsCap gates subscriptions
only — the WsCapMatcher is added just for subscribe requests, and chain-level SubMethods
advertisement — so a not-live upstream still serves all ordinary RPC.

What changed

New head-liveness cap detector

  • WsHeadLivenessCapDetector (internal/upstreams/caps/ws_head_liveness_detector.go), generic
    over the cap it governs. It selects over the WebSocket connector's SubscribeStates and
    the shared head processor's HeadEvent stream, asserting the cap only while the connector is
    WsConnected and the head is live.
  • Liveness reducer headLivenessTracker, ported from dshackle's HeadLivenessValidatorImpl: a
    head becomes live once it has produced 3 blocks in a row (2 consecutive height increments).
    Following dshackle, any diff <= 1 counts as consecutive — duplicate heights and backward
    reorgs are tolerated
    — and only a forward gap (diff >= 2) resets the run and starts a
    5-minute cooldown, during which the head stays not-live even after it resumes advancing.
  • Stall timeout: liveness is retracted when no head progress arrives within an adaptive
    window (measuredBlockTime × 3 × 2). measuredBlockTime starts at the chain's
    expected-block-time, grows to the largest observed inter-block interval, and doubles on each
    timeout (capped at 10 min). A socket drop is still caught independently by the connector-state
    gate, and the head processor's no-update watchdog still resubscribes.

Shared head processor

  • The HeadProcessor is now built once per upstream (CreateHeadProcessor) and shared between
    the head-event processor (which still owns its start/stop) and the cap detector (read-only
    Subscribe). caps.DetectorInput gains a Head field carrying it.

EVM wiring

  • EvmChainSpecificObject.CapDetectors gates WsCap with the head-liveness detector when the
    head connector is a WebSocket and validation is enabled; otherwise it keeps the plain
    WsPresenceCapDetector. The chain's expected-block-time is threaded into the detector to
    seed measuredBlockTime. Poll-head EVM upstreams and non-EVM chains are unchanged, and the
    NewHeadsCap / LogsCap / PendingTxCap detectors are untouched. EVM-only for now:
    consecutiveness assumes +1 height increments, which does not fit Solana's slot-based heads.

Config option

  • disable-liveness-subscription-validation *bool on chains.Options
    (yaml:"disable-liveness-subscription-validation"), settable in chains.yaml, in
    chain-defaults, or per upstream. Resolved (upstream → chain-defaults → chain settings) in
    setOptionsDefaults with a mode-dependent fallback (true in default mode, false in
    strict mode), plus a nil-safe accessor Options.LivenessSubscriptionValidationDisabled().

Config

upstream-config:
  chain-defaults:
    ethereum:
      options:
        disable-liveness-subscription-validation: false   # explicit; default is mode-dependent

true skips the check and restores the historical "connected WebSocket ⇒ WsCap" behavior.
Left unset, it defaults to true in default mode (gate off) and false in strict mode
(gate on).

Notes / impact

  • Warm-up: a freshly started ws-head EVM upstream serves RPC immediately but won't back
    subscriptions until its head has advanced 3 consecutive blocks — and, after any forward gap,
    for a further 5-minute cooldown.
  • Blast radius of a drop: losing WsCap removes the upstream as a target for all node-backed
    subscriptions (and de-advertises sub topics if every upstream on the chain loses it); it never
    affects plain RPC routing.

Docs

  • docs/nodecore/05-upstream-config.md — new disable-liveness-subscription-validation option.
  • docs/nodecore/13-subscriptions.md — new "Head-liveness gate on WsCap" note.

Testing

  • Reducer unit tests (ws_head_liveness_detector_internal_test.go, fake clock): block-count path;
    duplicate heights and backward reorgs advance the run; forward-gap reset; cooldown (stays
    not-live through a fresh run until the cooldown elapses); measuredBlockTime growth; onTimeout
    backoff + maxBlockTime cap; immediate recovery after a timeout.
  • Detector unit tests (ws_head_liveness_detector_test.go): asserted only when connected AND
    consecutive; disconnect and forward-gap retract; backward-reorg and duplicate-while-live keep it;
    a stalled head times out and retracts; nil connector / nil head never assert.
  • Config: mode-dependent default assertions (both default and strict) + updated full-config
    fixtures.
  • go build ./..., go test -race ./internal/..., and golangci-lint run all pass.

@Musthaq22 Musthaq22 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DSHACKLESIG

@KirillPamPam
KirillPamPam requested a review from a10zn8 July 23, 2026 09:11
@KirillPamPam
KirillPamPam merged commit 9728f16 into main Jul 23, 2026
5 checks passed
@KirillPamPam
KirillPamPam deleted the head-liveness branch July 23, 2026 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants