[Bug][Qwen3.5] Tracking: Qwen3.5 model issues with PD disaggregation

[YAMY1234]

Summary

Tracking issues encountered running Qwen3.5-397B-A17B-FP8 (Mamba hybrid MoE) in PD disaggregation mode. Most issues affect any Mamba hybrid model in disagg, some affect non-MLA GQA models more broadly.

Environment: SGLang dev (latest), GB200 (sm100), Qwen3.5-397B-A17B-FP8, Mooncake transfer backend.

Fixes with PRs

• Issue 1: Disagg decode ignores --max-mamba-cache-size → OOM ([Fix][Qwen3.5] Pass max_mamba_cache_size to mamba pool in disaggregation decode path #19002)
• Issue 2: int32 overflow in Mooncake send_kvcache_slice when KV buffer > 2GB ([Fix] Quick fix for int32 overflow in Mooncakes' send_kvcache_slice #19076, [Fix][Qwen3.5] Fix KV cache slice transfer for GQA models with replicated KV heads #19086)
• Issue 5: Division by zero & accuracy regression in send_kvcache_slice for GQA with few KV heads ([Fix][Qwen3.5] Fix KV cache slice transfer for GQA models with replicated KV heads #19086)
• Issue 6: Mamba cache ratio miscalculation when radix cache disabled → OOM at high concurrency ([Qwen3.5] Raise Exception when radix_cache and extra_buffer are enabled at the same time #19169)
• Issue 8: TP+TEP decode may crash when radix cache is disabled under high concurrency → KVTransferError (“failed to get kvcache from prefill instance, it might be dead”), likely bootstrap/prefill liveness or transfer coordination issue. Resolved: [Bug] PD Disaggregation: Mooncake NVLink transport fails with "Requested address not found" causing cascading KV transfer failures on Qwen3.5 with Concurrency>=512 #19355
• Issue 7: Issue 7: Qwen3-next accuracy issue, which may affect Qwen3.5, ACC & acclen is verified.
• Issue 9: query_dp_ranks connection failure → DP bootstrap HTTP endpoint intermittently unreachable with identical config; regression from recent networking/bootstrap change ([Bug] Qwen3.5 + pddisagg: Error querying dp_ranks from bootstrap in all disagg configs #19271). Fixed by [Disagg] Fix decode querying unregistered dp_rank when prefill dp_size is 1 #19305

Ongoing

• Issue 3: HybridLinearKVPool.get_cpu_copy NotImplementedError when disagg decode triggers retraction (draft fix, needs verification)
• Issue 4: DeepEP kNumMaxTopK=9 assertion failure for Qwen3.5 num_experts_per_tok=10 (upstream DeepEP issue, workaround: rebuild with kNumMaxTopK=16)

---

Issue 1: Disagg decode ignores --max-mamba-cache-size ✅ Fixed

PR: #19002

HybridMambaDecodeReqToTokenPool ignores --max-mamba-cache-size and always sizes the mamba pool at max_num_reqs. With extra_buffer strategy (3 slots/request), this causes immediate OOM. Affects all Mamba hybrid models in disagg decode.

---

Issue 2: int32 overflow in Mooncake send_kvcache_slice ✅ Fixed

PRs: #19076, #19086

KV cache byte offset computed with np.int32 overflows when buffer > 2 GB. Affects any non-MLA model with heterogeneous TP in disagg.

---

Issue 3: HybridLinearKVPool.get_cpu_copy NotImplementedError

Status: Draft fix, needs verification

When disagg decode triggers retraction (high token pool usage), release_req → offload_kv_cache → get_cpu_copy hits NotImplementedError in HybridLinearKVPool. Fix: delegate to internal full_kv_pool (MHATokenToKVPool); Mamba state not offloaded — retracted requests re-prefill to restore.

Trigger: disagg decode + Mamba hybrid model + high concurrency causing retraction.

---

Issue 4: DeepEP kNumMaxTopK assertion failure

Status: Upstream DeepEP issue

DeepEP low-latency kernel hardcodes kNumMaxTopK = 9. Qwen3.5 uses num_experts_per_tok = 10 → assertion failure. Workaround: rebuild DeepEP with kNumMaxTopK = 16.

---

Issue 5: Division by zero & accuracy regression in send_kvcache_slice for GQA ✅ Fixed

PR: #19086

num_kv_heads * attn_tp_size incorrectly back-computes total KV heads when head replication is involved (total_kv_heads < attn_tp_size), causing both division by zero and accuracy regression. Fix introduces total_kv_head_num to use the true un-sharded head count.

Affects: non-MLA GQA models with few KV heads in heterogeneous TP disagg.

---

Issue 6: Raise exception when radix is disabled but extra_buffer is enabled

PR: #19169

Issue 7: Qwen3-next accuracy issue, which may affect Qwen3.5

Instruct/Thinking Model Accuracy Test (GSM8K)
Prefix Caching:0.94/0.92
Prefix Caching + MTP:0.94/0.93
Disagg + Prefix Caching: 0.95/0.93
Disagg + Prefix Caching + MTP: 0.71/0.61 low accuracy!!!

GPQA Disagg + Prefix Caching
Repeat: 8, mean: 0.769
Scores: ['0.763', '0.793', '0.778', '0.798', '0.773', '0.768', '0.737', '0.742']
GPQA Reference
Repeat: 8, mean: 0.769
Scores: ['0.768', '0.763', '0.732', '0.798', '0.758', '0.773', '0.763', '0.798']

Issue 8:

Crash with TP & TEP decode when radixcache is off & concurrency is high:

Decode transfer failed for request rank=3 decode_req.req.rid='7eb1c2be0dd64ba3b8b745b452f0f120' 
decode_req.req.bootstrap_room=4434964693992720714 with exception 
KVTransferError(bootstrap_room=4434964693992720714): Failed to get kvcache from prefill instance, it might be dead

Issue 9:

#19271

Error querying dp_ranks from bootstrap: HTTPConnectionPool(host='10.66.4.40', port=38711): Max retries exceeded with url: /query_dp_ranks (Caused by NewConnectionError("HTTPConnection(host='10.66.4.40', port=38711): Failed to establish a new connection: [Errno 99] Cannot assign requested address"))

Using exactly the same config, the error does not occur before #19168

[hlu1]

cc @ispobock @ShangmingCai @yizhang2077 @hebiao064 @Qiaolin-Yu @Fridge003

[ShangmingCai]

@YAMY1234 For issue 2: Looks like I made a mistake, which didn't pop up during CI, when cleaning up code: afae4c7#diff-3be47cd55ac1c34f7566da855340b2ac3871526c03906247d9e7fe8609e6b7aeL467-L468
Feel free to make a quick PR to fix this.

For issue 5, maybe we can raise an error or return a failure, instead of using max(1, x)?

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.