Skip to content

fix: don't advance the book-version chain when no frame is emitted (#115)#116

Merged
emrebulutlar merged 1 commit into
mainfrom
fix/115-book-chain-empty-delta
Jul 8, 2026
Merged

fix: don't advance the book-version chain when no frame is emitted (#115)#116
emrebulutlar merged 1 commit into
mainfrom
fix/115-book-chain-empty-delta

Conversation

@emrebulutlar

Copy link
Copy Markdown
Member

Fixes #115.

Root cause

MarketPublisher.encodeBookSnapshot advanced lastPublishedBookVersion = collectedBookVersion unconditionally — even when encodeBookDelta returned 0 because the mutation happened below the visible top-20 depth (side version bumped, visible diff empty, nothing broadcast). The next visible delta then carried a fromVersion no consumer ever received, which the gateway correctly treats as a chain break (#96): book flagged stale, frozen until the next periodic resnapshot (#112).

This explains every piece of the live evidence:

  • side-specific missing versions (deltaBidV == haveBidV, ask versions skipping by 4) — deep ask-side churn from the sim;
  • match_dropped_market_msgs_total = 0 on all nodes — nothing was dropped, nothing was ever sent;
  • steady ~4 breaks/market/min in steady state — the rate of below-depth-20 churn;
  • "exactly one break per market right after the initial snapshot" (fix: chain-break diagnostics at default log level with per-side versions #114 diagnostics).

No raw-frame capture needed: the defect reproduces deterministically in a unit test (see below), failing with the exact live shape (delta from=2000 while the consumer holds 1000).

Fix

Advance lastPublishedBookVersion only when a frame was actually encoded. An unsent version gap is fine: the next delta spans it (visible state → visible state), so the emitted stream is chain-continuous by construction. Genuine shed (bounded-queue drop, counted) still breaks the chain at the gateway — which is the correct signal, recovered by the #112 resnapshot.

Audit finding fixed alongside (same family)

AppClusteredService.drainQueue gives up on a message for a session after 3 backpressured session.offer retries with no accounting anywhere — an egress loss invisible to every existing drop counter. Now counted as match_egress_offer_giveups_total (should stay 0) with a rate-limited CRITICAL log. Retry-policy hardening (esp. for the reliable OMS queue) left as a follow-up.

Tests

  • MarketPublisherBookChainTest.deepBookChangeMustNotBreakTheEmittedVersionChain — snapshot → deep-only change (no frame, by design) → visible change → delta must chain from the last emitted frame's version. Fails pre-fix with expected:<1000> but was:<2000>.
  • MarketPublisherBookChainTest.emittedStreamIsChainContinuousAcrossManyMixedFlushes — chain continuity across 10 interleaved deep/visible flush cycles.
  • Full match-cluster suite: 454 tests green.

🤖 Generated with Claude Code

)

Root cause of the steady-state chain flaps (~4/market/min, zero
cluster-side shed): a book mutation below the visible top-20 depth bumps
the side versions, so encodeBookSnapshot proceeds, but the visible diff
is empty — encodeBookDelta returns 0 and nothing is broadcast. Yet
lastPublishedBookVersion advanced anyway, so the next visible delta
named a fromVersion no consumer ever received. The gateway (correctly,
per match#96 drop-don't-drift) flagged a chain break and froze the book
until the next match#112 periodic resnapshot. Books were silently
drifting through these same skips before match#96 exposed them.

Fix: advance lastPublishedBookVersion only when a frame was actually
encoded. An unsent version gap is fine — the next delta spans it
(visible state to visible state), so the emitted stream stays
chain-continuous by construction.

Also (audit finding, same family): drainQueue() gave up on a message for
a session after 3 backpressured offer retries with NO accounting — a
real egress loss invisible to every existing drop counter. Now counted
(match_egress_offer_giveups_total, should stay 0) with a rate-limited
CRITICAL log.

Regression tests: MarketPublisherBookChainTest reproduces the exact live
break shape (delta fromVersion=2000 while the consumer holds 1000) and
asserts chain continuity across mixed deep/visible flush cycles.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@emrebulutlar
emrebulutlar merged commit aab39dd into main Jul 8, 2026
2 checks passed
@emrebulutlar
emrebulutlar deleted the fix/115-book-chain-empty-delta branch July 8, 2026 01:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway book chain flaps in steady state: continuous chain breaks (~4/market/min) with zero cluster-side shed

1 participant