fix: DirectIndexOrderBook requote exhausted the 64/level cap via unfreed tombstones (#94)

[emrebulutlar]

Mechanism

DirectIndexOrderBook.cancelOrder tombstones an order (sets qty=0) instead of unlinking it, keeping the slot in the singly-linked FIFO chain. A tombstone's slot is reclaimed only when:

1. the whole level empties (orderCount==0 triggers freeAllSlotsAtLevel), or
2. the lazy head-advance inside getHeadOrder* walks past it during a matching sweep at that price.

A price level pinned live by another user's order never trades, so its head never advances. Every cancel-then-replace at that price (Engine.processUpdate = cancel old + re-add new) permanently consumes one of the 64 MAX_ORDERS_PER_LEVEL slots. After ~62 requotes the level returns LEVEL_FULL despite only two orders genuinely resting. The default ArrayMatchingEngine has no per-level cap and rests them fine, so the direct engine (MATCH_ENGINE_IMPL=direct) diverges from array, breaking the A/B behavioral-parity invariant. Medium severity, fallback engine only.

Approach: compact on allocation pressure (not unlink-on-cancel)

The FIFO chain is singly linked (ORDER_FIELDS=4, no prev pointer), so O(1) unlink-on-cancel is impossible without a layout change (a prev pointer would grow every level's footprint). Instead, when addOrder finds the level's free stack exhausted, it calls a new compactLevel(priceIdx):

• walk the chain once (≤64 nodes), tracking the last surviving live node as we go;
• unlink and free every tombstone (qty==0), re-stitching live orders into a compact chain in their original FIFO order;
• rebuild head/tail; then retry the allocation.

LEVEL_FULL is now returned only if 64 genuinely-live orders remain after compaction.

Why this is safe:

• Slots do not move, so orderLocations entries for survivors stay valid.
• Every tombstone already had its orderLocations entry cleared (in cancelOrder/reduceOrderQuantity), so freeing its slot cannot orphan a lookup.
• Post-compaction the chain holds no tombstones, so the lazy head-advance in getHeadOrder* frees nothing further — no double-free.
• Compaction is a pure function of book state (no time/randomness), so determinism holds.
• The documented level_full_reject divergence (64 real live orders) is preserved.

Tests

DirectIndexOrderBookTest (4 new):

• testRequotePinnedLevel_NoLevelFull — issue trigger: B pins the level, A cancel-then-replaces 100×, every replace succeeds; only 2 orders rest.
• testLevelFull_64LiveOrders_CompactionCannotHelp — 64 genuinely-live orders → 65th still LEVEL_FULL (boundary preserved).
• testFifoPreservedAcrossCompaction — interleaved live/tombstones, forced compaction, survivors match in exact FIFO order.
• testCompactionNoDoubleFreeAndCleanNotFound — level totals/getActiveOrders consistent after 200 compaction cycles; a compacted-away id reports a clean not-found; level still fills to true 64 capacity.

Determinism suite:

• New requote_level_slots.scenario (B pins the $60k level, A requotes 70×) added to the convergent set — DeterminismAbEquivalenceTest now asserts direct == array for it (it would have diverged on the old code). Golden generated via -Dupdate.golden=true; no existing golden changed.

Full match-cluster suite (excluding the embedded-cluster/archive-housekeeping infra tests that need live Aeron): 396 tests, 0 failures.

Closes #94

🤖 Generated with Claude Code