fix(ws): refuse stale latest below TipHW (FOOS)#11
Merged
snowkide merged 11 commits intoJul 24, 2026
Conversation
TipHW alone still allowed eth_getBlockByNumber("latest") to fail-open to a
stale upstream block when re-fetch of the WS tip missed. Cache the newHeads
header before fan-out and prefer it in EnforceHighestBlock for header-only
requests so HTTP cannot regress below a tip already delivered on the pod.
Co-authored-by: Cursor <cursoragent@cursor.com>
xray — see through AI slop with deterministic architecture PR diff reviews |
Record the tip-source upstream id with the cached newHeads head, partition
eth_getBlockByNumber("latest") using TipHW so WS upstreams are tried first,
and pin EnforceHighestBlock re-fetch to that tip source instead of only
excluding the stale HTTP responder.
Co-authored-by: Cursor <cursoragent@cursor.com>
jleeh
changed the title
Tip ownership already lives on per-upstream SuggestLatestBlock/LatestBlock and partitionUpstreamsByLatestBlock. Keep TipHW partitioning for "latest" plus the header cache floor; remove the redundant upstreamId pin registry. Co-authored-by: Cursor <cursoragent@cursor.com>
jleeh
changed the title
Drop the newHeads header-cache / tip-source registry overbuild. Tip ownership already lives on per-upstream pollers (SuggestLatestBlock) and EvmLeaderUpstream; EnforceHighestBlock now UseUpstream-pins the concrete tip re-fetch to that leader when its LatestBlock covers TipHW. Co-authored-by: Cursor <cursoragent@cursor.com>
jleeh
changed the title
Availability checks and EnforceHighestBlock were calling PollLatestBlockNumber, which can reuse a debounced tip behind network TipHW (WS/Redis), falsely rejecting eth_call and fail-opening stale latest. Add PollLatestBlockNumberNow and use it on those paths; force-poll the leader before pinning the tip re-fetch. Co-authored-by: Cursor <cursoragent@cursor.com>
jleeh
changed the title
EnforceHighestBlock used pickHighestBlock, which fail-opened to a lagging "latest" when the concrete TipHW fetch returned null/error. That is the MultiNode FOOS trigger after WS newHeads already delivered the higher head. Re-fetch tip (leader pin, then unconstrained), accept only responses that meet the tip floor, and return an error instead of stale. Also fail-open the per-upstream availability gate when poller lags TipHW, and only skip enforcement for cached latest that already meets the tip. Co-authored-by: Cursor <cursoragent@cursor.com>
jleeh
changed the title
Cross-pod TipHW Redis push was async, so sibling pods could still serve a lower HTTP latest after WS newHeads advanced highestUserObservations, silently demoting MultiNode via FOOS. Publish TipHW before fan-out and refresh from Redis when local TipHW would skip EnforceHighestBlock. Co-authored-by: Cursor <cursoragent@cursor.com>
If TipHW advanced from a WS newHeads observation on this pod, HTTP
eth_getBlockByNumber("latest", false) must return that header instead of
hard-failing when concrete tip re-fetch cannot reach TipHW yet.
Co-authored-by: Cursor <cursoragent@cursor.com>
Fallback newHeads must not advance TipHW while primaries are up (same rule as poller aggregation). Also let the per-request fallback escape hatch fire when primaries return null/emptyish for a concrete tip block, so tip re-fetch can use public fallbacks instead of hard-failing. Co-authored-by: Cursor <cursoragent@cursor.com>
That path masked TipHW inflation / missing failover. Keep TipHW floor from primary WS only, refuse-stale when tip re-fetch misses, and escape to fallbacks on emptyish primary misses. Co-authored-by: Cursor <cursoragent@cursor.com>
Skipping TipHW for fallback WS while Ingest still fans those heads out caused MultiNode FOOS on matic. TipHW must cover every delivered head; tip re-fetch of a fallback-advanced TipHW relies on the emptyish escape hatch to reach public upstreams when primaries miss. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
latest:EnforceHighestBlockno longer usespickHighestBlockagainst the lagging response when the concrete TipHW re-fetch misses. That fail-open is the MultiNode FOOS trigger after WSnewHeadsalready delivered a higher head.EvmLeaderUpstream(force-poll if needed), then retry unconstrained; accept only responses that meet the tip floor; otherwise return an error.latestthat already meets TipHW.Test plan
TestHttpServer_GetBlockByNumberLatest_RefetchPinsEvmLeaderUpstreamTestHttpServer_GetBlockByNumberLatest_RefusesStaleFailOpenTestHttpServer_EvmGetBlockByNumber(incl. FailFast cases updated for refuse-stale)snowkide/erpc:ws-tip-http-floor-test4over prod DS (currently ontest3)No live RPC nodesdrops; bounce CL only if still stuck after eRPC is healthy