Concurrent tool-calling load kills the worker: duplicate pages in the full KV free list (root cause), surfacing as SWA-slot leak/double-free

[salekseev]

Summary

Under concurrent tool-calling load the engine dies within ~10-20 minutes. The proximate
symptom is SWA slot accounting, but the root cause is a duplicate-page double-free in the
full KV free list: free_slots persistently holds the same pages twice, so _allocate hands
one page to two owners.

AssertionError: SWA-slot leak/double-free: free(3868) + tree(8742) != capacity(26231)
  File "freetoken/scheduler/cache.py", line 557, in check_integrity
  File "freetoken/scheduler/scheduler.py", line 144, in run_when_idle
  File "freetoken/scheduler/io.py", line 82, in _recv_msg_single_rank
  File "freetoken/scheduler/scheduler.py", line 211, in overlap_loop
Backend supervisor: backend worker freetoken-TP0-scheduler exited
Backend worker is gone and cannot be restarted; stopping the API server

Evidence

Four probes added to a 0.1.2 tree (diagnostic only, ~30 lines), one 7-minute soak of concurrent
tool-calling traffic:

probe	what it counts	result
DIAG_DUPFREE	duplicate pages present in free_slots at allocation time	3598 events
DIAG_ORPHAN	alloc_swa overwriting a still-live full_to_swa_index_mapping entry	71
DIAG_UNPAIRED	CacheManager._free releasing a still-swa-mapped page	0
DIAG_LAZY	same check inside lazy_free_region's closure	0

The duplicate counts are stable and specific:

DIAG_DUPFREE free_slots holds 2125 duplicate page(s) (130162 entries, 128037 unique)
DIAG_DUPFREE free_slots holds 2125 duplicate page(s) (130098 entries, 127973 unique)
DIAG_DUPFREE free_slots holds 2125 duplicate page(s) (127918 entries, 125793 unique)

Exactly 2125 pages are in the free list twice, persistently. That accounts for every observed
failure:

• two owners for one page → the second alloc_swa overwrites the first owner's swa mapping, and
  that swa slot becomes neither free-listed nor tree-owned → free + tree < cap
  (free(3868) + tree(8742) != capacity(26231))
• a page counted by both a live request and the tree → tree > cap
  (free(1406) + tree(27992) != capacity(26229))

Concurrency is required because two owners only collide when two requests are in flight. A
single sequential session runs indefinitely: 748 requests, 553 tool calls, 14 minutes, zero
errors, engine healthy.

Also observed with a smaller window pool (--swa-full-tokens-ratio 0.07, 9176 tokens): the same
workload dies sooner and differently, RuntimeError: SWA pool exhausted: need 1669, have 1306
from alloc_swa, because a small pool runs out before check_integrity notices the mismatch.
Same bug, earlier symptom. (#202 is this presentation.)

Load that reproduces it

4 concurrent multi-turn tool-calling conversations, each: ~7.7k-token system prompt, tools
offered, 4 turns, tool results fed back, max_tokens 160. Ordinary agent traffic. Roughly 275
requests / 180 tool calls before the crash; the dup-free counter is non-zero within about two
minutes.

Paths ruled out, with citations

Recorded so this ground does not need re-covering:

• CacheManager._free (scheduler/cache.py) never releases a still-mapped page — 0 hits.
• Neither does lazy_free_region's closure, which shadows self._free for the whole of
  _process_last_data — 0 hits. Note a probe on the method alone misses this path entirely.
• SWARadixCache.evict_full correctly appends live-swa leaves to swa_indices
  (if not node.swa_tombstone: swa.append(node.value)).
• _cascade_swa_tombstone_leaves only walks nodes already swa_tombstone, whose swa slots were
  returned when they were tombstoned, so appending their full KV to kv alone is right.
• SWAEvictResult.swa_indices does hold full-pool indices, matching what free_swa expects —
  no unit mismatch.

That leaves the producer of the duplicates: some path returns a page to free_slots that is
still owned (by a request row or by the tree), or returns it twice. The comments around
_process_last_data and _free_req_resources show the hazard is already known in the
chunked-prefill/abort overlap ("cache_req double-frees the prior chunk", "double-freeing its
table_idx ... handing the same slots to two later requests"), which looks like the most likely
neighbourhood.

Suggested guard

A cheap invariant that fires in ~2 minutes under this load, long before the SWA assert, and
points straight at the allocation-side symptom:

assert self.free_slots.numel() == torch.unique(self.free_slots).numel()

in _allocate, or in check_integrity which already runs when idle.

Robustness points independent of the root cause

Each of these would downgrade this class of bug from an outage to a diagnostic, and they are
worth fixing whether or not the double-free is found quickly. (Folded in from #202, closed as a
duplicate of this issue.)

1. check_integrity failing terminates the worker. The assert is a diagnostic; dying on it
   converts a memory-accounting bug into an outage. Logging, or quarantining the affected slots,
   would keep the server serving.
2. alloc_swa raising on exhaustion is fatal rather than throttling. Everywhere else the
   scheduler treats capacity as a scheduling constraint — a request waits for KV pages. Window
   slots are the one pool whose exhaustion is an exception, so a transient shortfall of 363 slots
   ended the process (RuntimeError: SWA pool exhausted: need 1669, have 1306). Deferring the
   batch, or refusing the request with a 503, would keep the server up.
3. The advertised window-pool floor is not concurrency-aware. /v1/cache/status reported the
   pool as resizable down to 8777 tokens and --swa-full-tokens-ratio 0.07 resolved to 9176 —
   both accepted at startup without complaint, though 4 concurrent sequences each need a live
   trailing window plus the retain gap, and prefill needs its own slots on top. A floor
   accounting for max_running_requests * (window + _SWA_RETAIN_GAP) plus the prefill chunk
   would reject such a configuration at startup rather than crashing 9 minutes into production
   traffic.
4. Restart the backend worker on death, or fail /health immediately once it is gone. At
   present the frontend keeps answering /health and /v1/models for a while afterwards, so an
   operator sees a healthy port and hanging requests.

Environment

• FreeToken 0.1.2 (PyPI wheel + freetoken_kernel_cache-0.1.2+cu130)
• RTX 4080 SUPER (sm_89, 16 GB), driver 610.57.04 / CUDA UMD 13.3
• unsloth/gemma-4-26B-A4B-it-qat-GGUF (Q4_0), window 1024, 25 SWA / 5 full layers
• --kv-reserve-tokens 131072 --kv-cache-dtype q8_0 --max-running-requests 4 --swa-full-tokens-ratio 0.2
• Tree also carries fix(kernel): launch GGUF MoE GEMV token axis on grid.x, not grid.z #185, feat(kvcache): 8-bit KV cache (q8_0 / fp8_e4m3) behind --kv-cache-dtype #103, feat(server): expose swa_full_tokens_ratio as a startup CLI flag #109 and local gemma4 fixes for gemma4 GGUF: embedding quant type hardcoded to Q6_K, so a Q4_0-embedding GGUF fails to load with a bare AssertionError #188 / gemma-4 GGUF: <|tool_response> is not a stop token, so every tool call burns max_tokens and returns finish_reason=length #201; none touch page or
  slot accounting, and the dup-free counter is non-zero on stock defaults too.

Happy to run a candidate fix or a more targeted probe on this hardware — reproduction is
reliable in under 10 minutes.