Array-backed order book + egress hardening (v0.2.0-alpha)#27
Merged
Conversation
Same-input->same-output regression net for the matching engine. - MatchEventSink interface: capture engine output synchronously (no Disruptor) - SnapshotCodec extracted from AppClusteredService (byte format unchanged) - DeterminismCorpusTest: 15 .scenario/.golden pairs (golden + run-twice + wall-clock invariance) - SnapshotCodecTest + EngineSnapshotReplayTest: round-trip, byte-determinism, snapshot-transparent replay - MatchingInvariantsTest: seeded conservation + no-crossed-book + determinism - FixedPointTest: exactness/round-trip/commutativity cases - tools/durability: deterministic multi-node scenarios via Admin API 508 tests green (incl. real-Aeron EmbeddedClusterTest). No hot-path behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s, no 64-cap) Replace the preallocated DirectIndexOrderBook/DirectMatchingEngine with an array-backed, pooled, intrusive ordered index behind a new MatchingEngine interface, selected by match.engine.impl / MATCH_ENGINE_IMPL (default "array"; "direct" falls back to the preallocated engine). - ArrayOrderBook: AA tree of price levels in parallel primitive arrays (ported from AATree) + a shared pooled doubly-linked FIFO order list. Orders key on price (not node id) so AA delete-by-copy stays correct for external refs. Geometry-free, depth bounded only by a tunable pool (loud BOOK_FULL), zero hot-path allocation, no GC references. - ArrayMatchingEngine: drop-in match loop on level node ids; cross-thread top-of-book published on the writer thread via a seqlock (the tree cannot be walked lock-free like the positionally-stable direct book). - PriceRules: re-homed tick/range validation (range now runtime-adjustable), preserving exact existing rejection behavior. - MarketPublisher: chunk the trade-execution batch like the order-status batch (pre-existing fixed 256KB encodeBuffer overflowed on a >~3600-trade flush burst, e.g. egress re-emission on a leader takeover). Guarded by a MatchingEngine interface + flag so both impls run the determinism corpus: byte-identical except level_full_reject (64-cap removed -> 65th order rests as NEW; golden regenerated). Added A/B equivalence, cross-impl snapshot round-trip, model-stress-vs-TreeMap, heap-footprint, and trade-batch-chunking tests. Validated on the live 3-node cluster: ~580MB heap freed (848->267MB), latency parity (baseline p99 10.4us, stress p99 6.0ms), zero flush errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… under load
The egress pipeline had exactly one unbounded stage: MarketPublisher's per-event
OMS-bound buffers (tradeExecutionBuffer, orderStatusBuffer), cleared only by the
50ms flush. The disruptor ring upstream and the omsEgressQueue/marketDataQueue
downstream are both bounded, but these ArrayLists were not — so if the flush fell
behind (CPU starvation, a GC pause, or a slow/stalled egress consumer) they grew
without bound until the heap was exhausted and the matching/consensus threads OOM'd.
Reproduced by a sustained ~118k orders/sec load test: two nodes OOM'd (auto-restarted
and recovered via Raft, but a real production-disaster failure mode).
Backpressuring the matching thread isn't an option: the egress drain (flush()) runs
on the same cluster thread, so blocking it would deadlock. Instead, bound the buffers
and drop-with-loud-log on overflow — the same policy the egress queues already use.
Dropped settlement events are recoverable from the authoritative cluster log via OMS
reconciliation. Memory now caps at a few tens of MB; under overload the system sheds
load (Aeron ingress backpressure) instead of crashing. Cap tunable via
-Dmatch.egress.buffer.max (default 200k); exposed getDropped{Trade,Status}Events()
for monitoring. Pre-existing bug, independent of the array engine (whose freed heap
only delayed it). Regression test drives the buffer past a small cap and asserts it
drops rather than grows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eme load The MarketPublisher buffer fix (f45266f) closed one link of the egress chain, but the next link OOM'd: AppClusteredService's omsEgressQueue/marketDataQueue are bounded by ENTRY COUNT (262144 / 10000), yet each entry holds a copied batch buffer (up to ~150KB), so a backed-up egress consumer (or replaying a huge log) fills the heap and OOMs the matching/consensus threads. A 400k orders/sec load test crashed two nodes and broke recovery this way. Also cap total queued BYTES (omsEgress 128MB, marketData 32MB) — incremented on offer, decremented on drain; drop with a loud log on overflow. The egress is now byte-bounded end-to-end (ring -> MarketPublisher buffers -> these queues), capping egress heap at ~176MB regardless of consumer speed. Dropped settlement is recoverable via OMS reconciliation against the cluster log. Validated: the same 400k/sec overload that previously OOM'd the cluster now runs at 281k orders/sec, 100% success, ZERO OOM — gracefully shedding 0.08% of egress (loud CRITICAL log) instead of crashing; heap bounded, all 3 nodes healthy throughout. Throughput is HIGHER (281k vs 209k) because egress no longer destabilizes the cluster. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the preallocated order book with an array-backed, geometry-free engine, and bounds the egress pipeline so matching nodes can't OOM under load. Engine selected by
match.engine.impl/MATCH_ENGINE_IMPL(default flipped toarray;directfalls back to the old engine).This branch stacks on top of the determinism/durability suite. Review commit-by-commit:
41c3cf2test(cluster): determinism golden-master + snapshot/durability suite (the guardrail)de01971feat(cluster): array-backed order book (geometry-free, memory ∝ orders, no 64-cap)f45266ffix(cluster): bound the MarketPublisher egress buffers (entry cap)cff3fbefix(cluster): byte-bound the egress queues (eliminate OOM under extreme load)What changed
BOOK_FULL), zero hot-path allocation. Orders key on price (not node id) so AA delete-by-copy stays correct. Cross-thread top-of-book via a seqlock.omsEgressQueue/marketDataQueue, ~176 MB cap); sheds with a loud log + OMS reconciliation instead of crashing.Validated (single 13700K, 3-node Raft cluster, dev/local)
long[]735 → 182 MB live).Tunables / notes
MATCH_ENGINE_IMPL=directreverts to the preallocated engine.match.engine.book.capacity(default 131072/side),match.egress.buffer.max(default 200k).FixedPoint.multiplylarge-price overflow is preserved as-is (tracked separately).Alpha. Tag:
v0.2.0-alpha.🤖 Generated with Claude Code