Observed on hanzo-mainnet (36963), v1.36.57/58, 2026-08-06.
A node serving catch-up to a resyncing peer emits sent context response numBlocks=256 payloadBytes≈380k in an unbounded firehose — including sentTo=0 frames queued for a peer that is already gone — until the cgroup limit kills it (exit 137, OOMKilled). GOMEMLIMIT=4GiB under a 5Gi limit does not save it: the growth is buffer queueing, not GC-reclaimable heap.
Effect: every peer resync rotates through quorum members OOM-killing them (observed mv-0, mv-1 ×4, mv-3 ×8, mv-4 ×2 in one night), which repeatedly breaks quorum and — worse — an OOM landing mid-write corrupts the victim's stores (see companion issues).
Fix shape: bound the outbound context-frame queue, stop serving a disconnected peer, and backpressure the requester.
Single serve observed at count=48500 entries in one request (requestID 19371).
Observed on hanzo-mainnet (36963), v1.36.57/58, 2026-08-06.
A node serving catch-up to a resyncing peer emits
sent context response numBlocks=256 payloadBytes≈380kin an unbounded firehose — includingsentTo=0frames queued for a peer that is already gone — until the cgroup limit kills it (exit 137, OOMKilled). GOMEMLIMIT=4GiB under a 5Gi limit does not save it: the growth is buffer queueing, not GC-reclaimable heap.Effect: every peer resync rotates through quorum members OOM-killing them (observed mv-0, mv-1 ×4, mv-3 ×8, mv-4 ×2 in one night), which repeatedly breaks quorum and — worse — an OOM landing mid-write corrupts the victim's stores (see companion issues).
Fix shape: bound the outbound context-frame queue, stop serving a disconnected peer, and backpressure the requester.
Single serve observed at
count=48500entries in one request (requestID 19371).