fix: periodic book resnapshot bounds gateway staleness after a chain break#112
Merged
Conversation
…break The lossy market-data egress can shed a delta under burst (drop-oldest byte budget). Since match#96 the gateway then rightly stops applying deltas and marks the book stale, but recovery requires a fresh BOOK_SNAPSHOT and snapshots were only generated on session open, so a mid-session chain break froze the book until the next gateway reconnect. Observed live 2026-07-08 right after the match#96 deploy: one sim resume burst left all five books permanently stale (frozen UI books, moving tape). Reuse the existing resnapshot-generation machinery from the leader's flush timer: bump the generation every 10s while sessions exist. Each MarketPublisher re-sends a full snapshot; the gateway's match#95 monotonic guard drops equal-version duplicates, so the steady-state cost is a few small frames per market per interval, and a stale book now re-anchors within 10s. Egress-side only: nothing enters replicated state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Live regression found ~30min after deploying match#96's drop-and-flag-stale semantics: a sim resume burst shed deltas on the lossy egress, all 5 books went stale, and NOTHING ever delivered a recovery snapshot (they were only generated on session open) — frozen books, moving tape. Pre-#96 the same breaks were silently blind-applied, which is exactly the drift #96 fixed; the system always assumed a resnapshot path that did not exist mid-session.
Fix: bump the existing resnapshot generation from the leader's flush timer every 10s (sessions present). MarketPublishers re-send full snapshots; the match#95 equal-version guard swallows duplicates when nothing changed; a genuinely stale book re-anchors within 10s. Egress-side only, no replicated-state impact, no schema change.
Verified live after the roll (books under sim load recover from forced staleness within the interval; gateway_books_stale returns to 0).
🤖 Generated with Claude Code