Finding and solving issues with KV-cache management on Ninfer

[co-l]

Hello @Neroued

Thank you for your amazing work.

I spent the last week stress-testing ninfer on my setup (5090 + 32GB RAM), and noticed issues with cache management.

I saw you worked on the same issue recently, so I'm not sure you're interested in my work.

My goal is to maximize the capacity of ninfer under a real deployment, leveraging RAM as much as possible to avoid doing re-prefill. The motto is "never re-prefill an already prefilled context EXCEPT if eviction from both VRAM and RAM is inevitable"

cache-pressure protocol

Here is the bench protocol:

1. cache-pressure: verify the whole cache is usable, by filling it with 8K contexts and verify cache-hit in reverse order
2. agent-sim: simulates parallel sessions with subagents calls interleaved. 4x150K token "main" session are filled by 5K token increments, then for each 2 40K subagents are called, and then the main contexts are called again.
3. abort-sim: verify aborted session still get cache-hit
4. needle: verify needle is found in contexts

All these benches are available here: https://github.com/co-l/cache-pressure ; and directly callable with uvx as shown below

Ninfer launch config:

--kv-capacity 460000 --max-concurrency 4 --host-state-slots 96 --host-kv-mib 12288 --kv-dtype nvfp4 --spec mtp --draft-tokens 4

Results with current ninfer main

All benches against the same endpoint (--base-url defaults to
http://localhost:8000/v1; model auto-detected via GET /models).

1. cache-pressure: FAIL

# reproduce
uvx --from cache-pressure cache-pressure --base-url http://localhost:8000/v1 \
    --kv-size 480000 --output run.json

  [64/65] hydrate  ttft=   1.06s  e2e=   1.06s  prompt=  8017  out=   1  prefill= 7543tok/s  err=–  [512K/480K]
  [65/65] hydrate  ttft=   1.06s  e2e=   1.06s  prompt=  8012  out=   1  prefill= 7541tok/s  err=–  [520K/480K]
  actual cache used: 520,681 tokens (108.5% of capacity)

── Phase B: verify in reverse order (max_tokens=1, hit < 0.52s) ──
  [1/65] verify   ctx# 64  HIT   ttft=   0.01s  prompt=  8012  err=–  [8K/480K]
  [2/65] verify   ctx# 63  MISS  ttft=   1.85s  prompt=  8017  err=–  [16K/480K]
  first miss at ctx#63 — stopping (older contexts evicted by construction)

── Retention under pressure ──
  capacity:            480,000 tokens
  retained contexts:   1/65
  retained tokens:     8,012
  retained % capacity: 1.67%
  oldest evicted:      context #63 (older contexts evicted)

2. agent-sim: FAIL

# reproduce
uvx --from cache-pressure agent-sim --base-url http://localhost:8000/v1 \
    --sessions 4 --output agent-sim.json

  s3 main      step 23  prompt= 112,896  cached= 108,262  reuse=100.1%  ttft=4.03s  err=–
  s2 main      step 23  prompt= 112,994  cached= 108,360  reuse=100.1%  ttft=4.42s  err=–
  s0 main      step 24  prompt= 116,896  cached=       0  reuse=  0.0%  ttft=38.24s  err=–

(the following turns all re-prefilled from root, full cascade)

3. abort-sim: FAIL

# reproduce
uvx --from cache-pressure abort-sim --base-url http://localhost:8000/v1 \
    --runs 2 --salt 1789464054

abort-sim: 2 run(s) x ~40,000 tok context, abort after 500 streamed tokens, re-prompt max_tokens 1 (branch), min-reuse 95%, salt 1789464054
── run 1/2 · salt 1789464054 ──
  prime     ~40,000 tok context + problem · effort xhigh · budget 32,768
  abort     after 500 streamed tokens (500 reasoning / 0 content) · ttft 6.0s · wall 15.1s
  re-prompt prompt=38,664  cached=0 (usage)  reuse=0.0%  wall 6.4s
── run 2/2 · salt 1789564054 ──
  prime     ~40,000 tok context + problem · effort xhigh · budget 32,768
  abort     after 500 streamed tokens (500 reasoning / 0 content) · ttft 6.0s · wall 15.2s
  re-prompt prompt=38,289  cached=0 (usage)  reuse=0.0%  wall 6.4s

── verdict ──
  [FAIL] run 1: reuse 0.0% < 95%
  [FAIL] run 2: reuse 0.0% < 95%
  runs passed: 0/2

4. needle: PASS

# reproduce (full sweep 50K -> 450K)
uvx --from cache-pressure needle-test --base-url http://localhost:8000/v1

Notes on the commands (matching each run's parameters):

• cache-pressure: --kv-size 480000 from "capacity: 480,000"
• agent-sim: --sessions 4 matches s0–s3; main/sub sizes are the defaults (150K/40K), step 5K matches the step increments.
• If your endpoint isn't on localhost:8000, swap --base-url accordingly.

Tested against upstream master at commit d4929686 (perf(runtime): improve materialization search and adapt planning budgets, 2026-09-11) — master tip at test time. Note: master has since moved on to the v3 artifact work (20 commits past d4929686), which I have not tested.

My ninfer fork

Fork co-l/ninfer@legacy/validated-main-b2d27c8c, main at b2d27c8c (= your d4929686 + the cache-pressure fix work). Same box, same endpoint, same commands as above, run back-to-back on one server instance (dirty cache — no restart between benches).

1. cache-pressure: PASS

uvx --from cache-pressure cache-pressure --base-url http://localhost:8000/v1 --kv-size 480000

  actual cache used: 520,681 tokens (108.5% of capacity)
── Phase B: verify in reverse order (max_tokens=1, hit < 0.53s) ──
  [1/65] verify   ctx# 64  HIT   ttft=   0.01s  prompt=  8012  err=–  [8K/480K]
  ...
  [65/65] verify  ctx#  0  HIT   ttft=   0.05s  prompt=  8009  err=–  [520K/480K]
── Retention under pressure ──
  capacity:            480,000 tokens
  retained contexts:   65/65
  retained tokens:     520,616
  retained % capacity: 108.46%
  oldest evicted:      none (everything still cached)

2. agent-sim: PASS

uvx --from cache-pressure agent-sim --base-url http://localhost:8000/v1 \
    --sessions 4 --main-tokens 150000 --sub-tokens 40000

  main continuation turns reused: 116/116 (100%)
  FINALIZE survival (main context back from cache after subs): 4/4 (100%)
  sub windows cold (sanity): 8/8 (100%)

Ground truth from the request log: all mains/finalize served via private_endpoint, zero root re-prefills (only the 4 cold starts + 8 cold sub windows).

3. abort-sim: PASS (re-run on the dirty cache after cache-pressure saturated it)

uvx --from cache-pressure abort-sim --base-url http://localhost:8000/v1

── run 1/2 · salt 1789464054 ──
  prime     ~40,000 tok context + problem · effort xhigh · budget 32,768
  abort     after 500 streamed tokens (500 reasoning / 0 content) · ttft 0.2s · wall 9.4s
  re-prompt prompt=38,571  cached=37,225 (usage)  reuse=96.5%  wall 0.5s
── run 2/2 · salt 1789564054 ──
  re-prompt prompt=38,410  cached=37,110 (usage)  reuse=96.6%  wall 0.5s
── verdict ──
  [OK] run 1: reuse 96.5% ≥ 95%
  [OK] run 2: reuse 96.6% ≥ 95%
  runs passed: 2/2

4. needle: PASS

uvx --from cache-pressure needle-test --base-url http://localhost:8000/v1 \
    --lengths 50000,100000,200000

  50K: PASS · 100K: PASS · 200K: PASS — 3/3 needles found

It's a large change so I prefer not to push a PR against a moving target.

But the work is there and it's making a much stable version of the KV cache management.

If you decide to use this work, no attribution needed.

Tell me what you think.

PS: on my main branch, I also added tool streaming

[FlyToha]

Hey! Its nice fork! Can you please make windows version?