[Bug]: GLM-5.1-FP8 produces corrupted output under tool-use on H200 (deepseek_v3.2 FP8 path)

[KyleShao1016]

System Info

• GPU: 8× NVIDIA H200 (SM90)
• TensorRT-LLM: 1.3.0rc16 (also reviewed rc17/rc18 changelogs; no relevant fix landed)
• Container: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc16
• Model: zai-org/GLM-5.1-FP8 (model_type glm_moe_dsa, arch GlmMoeDsaForCausalLM; served via DeepseekV3ForCausalLM)
• Quant: FP8 e4m3, block-scaled weights (weight_block_size [128,128]), dynamic activation; no KV-cache quant declared
• Serving: trtllm-serve, TP8 / EP8, PyTorch backend, moe_config.backend=CUTLASS, MTP off,
  --tool_parser glm47 --reasoning_parser deepseek-r1, kv_cache_config.dtype=auto

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Serve GLM-5.1-FP8 on 8×H200:

trtllm-serve zai-org/GLM-5.1-FP8 --backend pytorch --tp_size 8 --ep_size 8 \
  --tool_parser glm47 --reasoning_parser deepseek-r1 \
  --kv_cache_config '{"dtype":"auto","enable_block_reuse":true}' \
  --moe_config '{"backend":"CUTLASS"}'

Send a chat request that contains real function-tool definitions inside a large system
prompt (~7–8k tokens of agent/policy text + 3 simple function schemas):

POST /v1/chat/completions
{
  "model": "zai-org/GLM-5.1-FP8",
  "messages": [
    {"role":"system","content":"<~7–8k-token agent policy>"},
    {"role":"assistant","content":"Hi! How can I help you today?"},
    {"role":"user","content":"<a task that requires a tool call>"}
  ],
  "tools": [ <3 function schemas, simple string params, with descriptions> ],
  "tool_choice": "auto",
  "temperature": 0,
  "max_tokens": 1024,
  "chat_template_kwargs": {"enable_thinking": true}
}

Control (clean): the same request with the tools' description fields removed, or with 0 tools,
or with a small (<2048-token) system prompt.

Expected behavior

A coherent response and/or a structured tool_calls entry

actual behavior

The model degenerates: finish_reason="length" (never emits a stop token, runs to max_tokens),
tool_calls=[], and the generated text is incoherent — repetitive/random tokens, digit and https-heavy fragments, occasionally mixed-language.
Example output (greedy):

<arg_value>,https,2hat, the15|35ions,16#51Obn2247777735'170044https,720c523722|422 to>
... d46aining is_pn4601ed ... 7771https17https_n7ar4#U29n60

additional notes

We narrowed this down and believe it is FP8 numerics on the deepseek_v3.2 path, not config:

• Deterministic: reproduces at temperature=0 (greedy) → not sampling.
• Logits collapsed: logprobs are flat from the first generated token (top token ~7%,
  whole top-5 tiny) → saturated/degenerate logits.
• Cross-engine: same checkpoint serves coherently on SGLang/vLLM → not the weights.
• Not KV-cache FP8: with dtype=auto the checkpoint declares no kv_cache_quant_algo, so the
  main KV cache is bf16 (confirmed: allocated 110,448 B/token ≈ the bf16 figure, ~2× the fp8 figure;
  forcing dtype=bfloat16 is rejected — only fp8/nvfp4/auto accepted). MoE backend is CUTLASS.
• Not dispatch/RoPE: rc16 renames glm_moe_dsa→deepseek_v32 before model build, so the DSA
  attention path is selected; the config has no rope_scaling and the prompt is well within
  max_position_embeddings (202752).
• Not the tokenizer: all GLM special tokens encode atomically; chat template renders with tools
  and round-trips.
• Trigger: real tool descriptions in a large context (>~2048 tokens, i.e. the DSA sparse
  regime). Small/toolless prompts are (mostly) clean.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[CodersAcademy006]

Hey @KyleShao1016 Thanks for the very thorough isolation work here, this narrows things down nicely. A couple of observations from digging into the DSA architecture and recent TRT-LLM history that might help narrow it further.

The >2048 token trigger lines up with a known architectural seam. DeepSeek-V3.2 uses K=2048 for the indexer's Top-K selector. For sequences ≤2048 tokens, all past KV tokens are inherently selected, so the indexer/Top-K path is bypassed entirely via a dense fast path (PR-9524). Above 2048, the actual lightning-indexer + Top-K selection kicks in. Your clean-vs-broken boundary maps directly onto this seam, not onto context length in general, which fits with everything else you've already ruled out (KV cache, dispatch/RoPE, tokenizer, weights).

The indexer's FP8 path has a documented precision history. The indexer Q/K projections went through blockwise FP8 quantization (PR-8701), and a later PR (PR-9243) had to move the indexer weight projection GEMM back to FP32 specifically to fix an accuracy regression. So this is a component that's already needed one FP8-calibration correction.

One thing I'd rule out first: the recently merged heuristic-guided indexer TopK (PR #12385) is gated to SM100+ (Blackwell) and disabled by default, so it shouldn't be in play on H200/SM90 here — just flagging in case it comes up as a red herring.

Hypothesis: GLM-5.1-FP8 is served through the deepseek_v3.2/glm_moe_dsa path, and the indexer's FP8 block-scale calibration may have been tuned against DeepSeek-V3.2's activation statistics rather than GLM's. Below 2048 tokens this never surfaces (indexer bypassed). Above 2048, with a long agent/tool-heavy prompt producing activation distributions outside the calibration range, the FP8-quantized indexer scores could saturate or collapse, causing Top-K to select near-random KV blocks, sparse attention to attend to noise, and logits to go flat from the first generated token, deterministic at greedy. This would also explain why other engines stay coherent (different indexer FP8 implementation/calibration).

A few cheap things that would help confirm or rule this out:

• Sweep prompt length around the 2048 boundary (e.g. ~2000 vs ~2100 tokens, same tool definitions) to confirm the fast-path/sparse-path seam is the exact trigger.
• If there's any way to force the indexer Q/K projections to bf16/fp32 (similar in spirit to the PR-9243 fix), re-run the >2048 case. Coherent output would isolate this to indexer FP8 quantization.
• If feasible, run an actual DeepSeek-V3.2-FP8 checkpoint through the same build with the same long tool prompt. If DeepSeek stays coherent and GLM doesn't, that points specifically at GLM's indexer config/calibration on the shared code path rather than a universal DSA issue.
• If accessible, dumping the dequantized lightning-indexer scores for a token past position 2048 vs one before would show whether FP8 saturation (clipping near e4m3 max, or near-identical scores) is actually happening.

[KyleShao1016]

Thanks @CodersAcademy006 — your DSA/indexer breakdown gave us exactly the right things to test. We ran three experiments on the same setup (GLM-5.1-FP8, release:1.3.0rc16, 8×H200, TP8/EP8, PyTorch backend, moe_config.backend=CUTLASS, kv_cache_config.dtype=auto → bf16 KV, greedy temp=0).
Summary: the 2048 seam isn't the trigger, the bug is GLM-specific, and it looks upstream of the indexer.

1. Length sweep — the trigger is NOT the K=2048 indexer seam

Took the known-broken request (3 real tools) and swept prompt_tokens across 2048 by truncating the system prompt, greedy:

• Coherent well past 2048 — clean at 2080, 2465, 4421, 5195 tokens.
• Onset ~5.4–5.9k tokens (broke at 5543 / 5689).
• Content-agnostic: a neutral generic-filler prompt (no domain text) reproduces the exact …<arg_value>…https… corruption at ~5.9k tokens.
• Requires tools: at matched ~5.9k length, 3 tools → corruption, 0 tools → coherent.

So the trigger is tool-definition tokens + total context beyond ~5.5k, not crossing the dense→sparse seam at 2048. (Consistent with an FP8 issue that worsens as the candidate set grows past 2048, rather than a sharp toggle.)

2. DeepSeek-V3.2 cross-model control — the corruption is GLM-specific

Served the native deepseek-ai/DeepSeek-V3.2 (FP8, deepseek_v32, same DSA index_topk=2048, bf16 KV) on the identical build/preset, with the repo's tool_chat_template_deepseekv31.jinja + --tool_parser deepseek_v32.
Same request shape, greedy:

• Coherent at 5,784 tokens with 3 tools (finish=stop), and still coherent at ~11k tokens with tool defs in context.

DeepSeek-V3.2 stays coherent exactly where GLM-5.1-FP8 collapses, on the same code path. ⇒ not a universal DSA/TRT-LLM defect — specific to GLM-5.1-FP8 (consistent with the FP8 block-scale calibration fitting DeepSeek's activation stats but not GLM's).

3. Indexer-score dump — the indexer is NOT where it breaks (so look upstream)

Instrumented _call_mqa_logits in dsa.py to dump lightning-indexer stats (marker-gated). Confirmed the ≤2048 dense bypass (1980-tok prompt → 0 indexer calls). Then compared a coherent >2048 (3.3k) run vs the broken (5.7k) run, both indexer-active, each metric averaged over 1296–2383 valid query rows (top-2048 gap):

signature	coherent 3.3k	broken 5.7k
top1−2048th score gap (collapse)	mean 21.9	mean 48.2 (discriminates more)
q/k median |·| (underflow)	56 / 44	48 / 64 (not crushed)
FP8 distinct levels used	253	253 (full e4m3 range both)
q/k max |·| (clipping)	448 / 448	448 / 448 (normal dynamic max)

So in the broken run the indexer scores discriminate at least as well as in the coherent run, and the q/k inputs use the full FP8 range with healthy magnitudes — no collapse, no underflow, no abnormal clipping. The indexer subsystem is numerically healthy even when the output is garbage.

Scope: this rules out the indexer FP8 scoring as the corruption site; it does not rule out FP8 issues upstream (the indexer can score confidently on q/k that already encode corrupted hidden states). So we read the corruption as upstream of the DSA indexer — FP8 MoE or main MLA attention the leading candidates. Pinning the first-diverging layer/op needs layer-wise activation tracing on your side.

Where we think this leaves it

A GLM-5.1-FP8-specific FP8 numerical problem on the DSA path, upstream of the lightning indexer, surfacing only with tool-definition tokens in a >~5.5k-token context, deterministic at greedy. DeepSeek-V3.2 is fine on the same code path, so it's not a universal DSA defect.

[CodersAcademy006]

Hey @KyleShao1016, thanks for running those experiments — the indexer dump is exactly the right thing to check and the results are unambiguous. You're right to walk back the K=2048 seam hypothesis; the ~5.5k onset with a content-agnostic filler prompt is a cleaner signal than a sharp architectural toggle would produce, and the discriminating indexer scores in the broken run rule out FP8 saturation at that layer.

The DeepSeek cross-model control is the most important result here. Same code path, same DSA configuration, same context length, coherent for DeepSeek, broken for GLM. That pins this firmly to GLM-5.1-FP8's activation statistics diverging from what the shared FP8 calibration expects, not a universal DSA defect.

Given the indexer is healthy, the two upstream candidates worth instrumenting are the FP8 MoE dispatch (specifically whether GLM's expert activation patterns at >5.5k tokens push router logits or expert hidden states outside the calibrated e4m3 range) and the main MLA attention FP8 path (whether the Q/K/V projections degrade at long context under GLM's weight block scales). A practical next step would be checking whether forcing the MoE GEMM or MLA projection GEMMs to bf16 on the DSA path restores coherence past 5.5k with tools, similar in spirit to the PR-9243 indexer fix, since that would isolate which of the two is the first diverging site without needing full layer-wise activation tracing.

[KyleShao1016]

Thanks — really helpful framing. I ran both bf16-GEMM tests you suggested. Short version: it's neither the MoE GEMM nor the MLA projection GEMMs.

Method: patched FP8BlockScalesLinearMethod.apply to dequantize the block-scale weights and run a bf16 F.linear instead of the fp8 GEMM, env-scoped, with logging to confirm exactly which linears fired. Ran on the VANILLA MoE backend so the experts route through the Linear path (CUTLASS's fused MoE bypasses it). Validated on the coherent (<2k) case each time — it stayed coherent, confirming the dequant is correct.

Results (all confirmed-fired via the shape log):

• bf16 MoE expert GEMM (scope=moe — log confirms it fired on the real expert shapes (4096,6144), (6144,2048)): coherent clean, broken (>5.5k + tools) still garbles.
• bf16 ALL block-scale linears (scope=all — MoE experts + every MLA Q/K/V/O projection: (2048,2048), (2624,6144), (3072,6144), (512,6144), (6144,1536), …): broken still garbles.

So bf16-ing both candidate sites together doesn't restore coherence — the first diverging site is not the block-scale GEMMs. (Router logits are already bf16 in the checkpoint — mlp.gate is in modules_to_not_convert — so never fp8.)

By elimination that loops back to the DSA indexer FP8 — which I know contradicts the "indexer is healthy" read, so to reconcile: the indexer scores discriminate fine (healthy top-1-vs-2048th gap, as we both saw) — but that's score discrimination, not selection correctness. FP8 q/k can confidently select the wrong top-2048. I also tested the #13061-spirit fix (Hadamard rotation on the indexer q/k before fused_cat_fp8, confirmed fired) and it did not help — so it's not the q/k quant quality; the remaining suspects are the fp8 indexer K-cache and the fp8_mqa_logits accumulation.

One more data point: per-layer hidden-state traces (both attn and moe sub-blocks) show no NaN/Inf/overflow/underflow at any layer, in prefill and decode — magnitudes stay bounded (broken ≈ coherent). So it's subtle wrong-values, not a numerical blow-up — consistent with the indexer selecting the wrong tokens → attention on wrong context → flat logits.

These are just my current thoughts based on the experiments above. If I'm misunderstanding anything about the indexer path or drawing the wrong conclusion from the results, please feel free to correct me.

[WeiHaocheng]

Hi @KyleShao1016, I tried to reproduce the bug with a prompt about 8k tokens but I can not reproduce it yet.

[REPRO] Using model path: /code/llm-models/GLM-5-FP8
[REPRO] system prompt: 33191 chars (~8297 tokens approx)
[REPRO] wrote config: /code/tensorrt_llm/build/repro_config.yaml
[REPRO] Launching: trtllm-serve /code/llm-models/GLM-5-FP8 --backend pytorch --host 127.0.0.1 --port 35975 --tp_size 8 --ep_size 8 --tool_parser glm47 --reasoning_parser deepseek-r1 --extra_llm_api_options /code/tensorrt_llm/build/repro_config.yaml

View Script
View Full Log
Test Output (passed)
[REPRO] Using model path: /code/llm-models/GLM-5-FP8
[REPRO] system prompt: 33191 chars (~8297 tokens approx)
[REPRO] wrote config: /code/tensorrt_llm/build/repro_config.yaml
[REPRO] Launching: trtllm-serve /code/llm-models/GLM-5-FP8 --backend pytorch --host 127.0.0.1 --port 35975 --tp_size 8 --ep_size 8 --tool_parser glm47 --reasoning_parser deepseek-r1 --extra_llm_api_options /code/tensorrt_llm/build/repro_config.yaml
[REPRO] Server log: /code/tensorrt_llm/build/repro_server.log
[REPRO] server is healthy after 510s
[REPRO] POST http://127.0.0.1:35975/v1/chat/completions  (system_prompt len=33191 chars; 3 tools)
[REPRO] finish_reason = 'tool_calls'
[REPRO] usage = {'prompt_tokens': 7414, 'total_tokens': 7541, 'completion_tokens': 127, 'prompt_tokens_details': {'cached_tokens': 0}}
[REPRO] #tool_calls = 1
[REPRO] content (first 400 chars): ''
[REPRO] reasoning_content (first 200 chars): "The user is asking for the current weather in San Francisco, California. They've explicitly asked me to use the appropriate tool to look it up. I have access to the get_weather tool which is perfect f"
[REPRO]   tool_call: name='get_weather' args='{"location": "San Francisco, California"}'

Could you give me the system prompt you used?

[lucifer1004]

We believe we have root-caused a failure mode that matches this issue, including why the reproduction attempts above fail. We hit it while validating a DeepSeek-V4 configuration with enable_block_reuse: true (the FLASHINFER block-reuse workaround lifted experimentally); the faulting machinery is the DSA indexer K-cache path shared by deepseek_v3.2/GLM and DSv4 (tensorrt_llm/_torch/attention_backend/sparse/dsa.py), so the backend and model specifics should not matter.

Mechanism

With block reuse enabled, completed requests' blocks in the indexer K-cache pool are retained for reuse but never recycled under sustained traffic, so the pool's base page indices grow monotonically with served volume. _compute_slot_mappings (dsa.py) computes

fp8_indices = block_ids * block_stride    # block_ids = base_page_index * scale

with no bound against the gather view, and once retention pushes block_ids past the pool extent, indexerKCacheGatherUnifiedKernel gathers out of range (cpp/tensorrt_llm/kernels/indexerKCacheGather.cu; the launch-check at :159 is where the async error surfaces when the reads finally leave mapped memory).

The key property: an out-of-range gather that still lands in mapped memory does not crash — it silently returns garbage indexer logits, so top-k selects wrong KV and the output is corrupted-but-fluent. Only under enough retention pressure do the reads hit unmapped pages and turn into cudaErrorIllegalAddress. We observed both stages directly.

Measurements (1.3.0rc21-based source; mechanism unchanged on main as of d72425e78d)

Instrumenting block_ids.max() in _compute_slot_mappings:

condition	indexer page index high-water	result
single ~7k-token request, fresh server	~651	fine
reuse ON, 5 rounds x 20 concurrent unique ~7k prompts	651 → 34839 → 35427 → 36036	CUDA illegal memory access
reuse OFF, identical load	stabilizes ~11739 (pages recycle)	5/5 rounds clean

Why this fits this issue

• Your serving config sets enable_block_reuse: true explicitly, and tool-use is exactly the reuse-retention-pressure workload (many sequential turns, most tokens retained).
• It explains the failed repro above: a fresh server + one long prompt has no retention pressure — the pool is nowhere near its extent. The corruption needs served-volume history first (many prior requests, or sustained concurrent unique-prompt load), then the affected request.
• It is consistent with corruption-without-crash on your setup (mapped-memory OOB reads → garbage indexer scores → wrong top-k), and with your earlier findings that the MoE GEMMs and MLA projection GEMMs are not involved.
• The ~5.5k onset you isolated plausibly tracks pressure growth rate rather than an architectural seam: longer contexts allocate more indexer pages per request, reaching the pool extent sooner.

Suggested repro for your setup

1. Serve with enable_block_reuse: true.
2. Pre-load with N rounds of unique long prompts (we used 5 rounds x 20 concurrent x ~7k tokens; scale N up until the watermark approaches the pool extent).
3. Run the tool-use conversation. Corruption probability rises with the watermark; the hard crash is the end state.
4. One-line instrumentation: log block_ids.max() in _compute_slot_mappings (dsa.py) per step and watch it grow monotonically with reuse ON vs plateau with reuse OFF.

Fix direction

The indexer K-cache (and the analogous compressed-KV) pools need reuse retention that stays within the gather's addressable extent: either make retained blocks evictable-and-recycled under pool pressure so base page indices stay bounded, or size/bound the gather view to the full physical pool and cap allocation. Bounding block_ids in _compute_slot_mappings alone is only an assert-grade stopgap — clamping silently gathers wrong KV, which is this issue's symptom, not a fix.

Happy to share the probe scripts (retention watermark logger, load driver, reuse on/off A-B harness) if useful.

[lucifer1004]

Update: we have root-caused and fixed the crash on our side, and the mechanism turned out to be narrower than my previous comment suggested — correcting the record.

The defect is a stride bug specific to compress-ratio>1 models (DeepSeek-V4): the metadata freezes the raw tokens_per_block before the DSv4 subclass installs its compress ratios (a super().__post_init__ ordering issue), while the indexer K-cache pages are compressed-native — so the fullkv/chunked-prefill gather walks compress_ratio×-inflated flat offsets and overruns the pool once block ids climb past capacity/compress_ratio. Block-reuse retention is what pushes block ids that high, which is why the failure is retention-pressure-dependent. Fix: #16223 (derive the indexer block stride from live manager geometry at use time, plus an env-gated bounds guard for the gather). With it, reuse-enabled DSv4 passes the full qualification including the concurrency ramp that previously faulted, and a 128k/512k/1M long-context ladder with prefix reuse. The DSv4-on-SM120 enablement that surfaced all of this is #16224.

What this means for this issue: GLM-5.1 / deepseek_v3.2 have compress divisor 1, so this exact stride mismatch does not apply to your setup — my earlier suggestion that it directly explains your corruption was too strong. Two parts of the analysis do still stand and may help: (1) your failure signature remains retention-pressure-dependent, so a fresh-server single-prompt repro will keep failing — priming the server with sustained unique-prompt traffic before the tool-use conversation should reproduce it; (2) the gather path has no bounds check in the C++ op (the guard added in #16223 covers it) — running with TRTLLM_DEBUG_INDEXER_GATHER_BOUNDS=1 on a build containing that PR would quickly tell whether your corruption is also an out-of-range gather (a sibling mechanism with a different index-side cause) or something else entirely.

[lucifer1004]

Following up with a negative reproduction result that narrows the remaining suspect surface from a different angle.

Setup

We recently enabled the DSA path on SM120 via flashinfer's sparse-MLA kernels (#16224). The indexer machinery — fp8 K-cache writes, fp8_paged_mqa_logits, top-k selection — is the same Python/DeepGEMM code this issue's config runs, just compiled for SM120; only the MLA attention core differs. That makes it a testbed for exactly the two suspects left standing after your eliminations (fp8 indexer K-cache, fp8_mqa_logits accumulation): if either is wrong as an algorithm or format, it should be wrong here too.

Config parity where possible: TP8/EP8, --tool_parser glm47 --reasoning_parser deepseek-r1, greedy, thinking enabled, enable_block_reuse: true (+ partial reuse), MTP off, rc21-line build, 8× RTX PRO 6000 Blackwell. Models: GLM-5.1-NVFP4 and GLM-5.2-NVFP4 (nvidia modelopt quants; attention and the entire indexer stay bf16 in these checkpoints — comparable to your all-bf16-GEMM control, which still garbled; the MoE experts differ, NVFP4 vs FP8, so hidden-state trajectories are not bitwise-transplanted).

Result: no corruption, 92 requests

Two methodologies, both arms:

• Independent rungs, 3 real tool schemas with descriptions + neutral filler, 2.3k→7.6k prompt tokens crossing the ~5.5k onset, tools vs no-tools at each rung: 16/16 clean, every tools arm a correct tool_calls.
• Your truncation methodology: one ~10.8k-token structured agent-policy prompt, ascending + descending truncation ladder against the reuse-enabled server (partial prefix reuse >96% on consecutive requests), each length asked twice (cold + reuse hit): 30/30 clean per model, cold/hit answers stable.

What we think this narrows

1. The surviving indexer suspects are unlikely to be format/algorithm defects. The same K-cache layout (fp8 + per-128 fp32 scales), the same slot-mapping code, and the same MQA-logits algorithm ran against GLM's own weights and stayed clean through 10.8k tokens with tools. What our test cannot exercise is the SM90 kernel binaries themselves (DeepGEMM's SM90 fp8_paged_mqa_logits/scheduling, the SM90 sparse-MLA integration) — and any explanation there has to be an interaction with GLM's activation statistics, since DeepSeek-V3.2 stays coherent on the identical binaries.
2. "Fixed in a newer rc" looks unlikely but is cheap to check. We reviewed every commit touching dsa.py/DeepGEMM/the MLA module between rc16 and rc21: only an MTP DSL guard and a KV-size estimation fix — no numerics change. Still, a one-off retest on a current rc would close that branch for good.
3. The missing artifact is still the exact prompt. Our synthetic reconstructions of the described trigger (generic filler + 3 described tools past 5.5k) did not corrupt even once across two model generations — consistent with @WeiHaocheng's non-repro at 7.4k. Whatever property of the original system prompt matters, neither of us has reproduced it from the description.

A discriminator we can contribute

Your framing — "score discrimination ≠ selection correctness" — suggests the decisive experiment on your setup: an env-gated shadow compare that, per layer, recomputes the top-2048 from bf16-dequantized indexer K alongside the production fp8 path and logs the selection overlap for the broken request. Healthy overlap ⇒ selection is correct and the corruption is downstream of the indexer after all; overlap collapsing past ~5.5k on GLM but not DSv3.2 ⇒ pinned to fp8-K selection under GLM's statistics on SM90. We've been in this code recently and are happy to prepare that patch against a recent rc if it would help.

[elmehdi-eljair]

Thanks for the exceptionally thorough isolation work here. We’ve been looking at this from the perspective of MemForge, our measurement-guided GPU diagnostics/optimization project:

https://github.com/elmehdi-eljair/memforge

At this point I agree that looking only at activation magnitude, FP8 range utilization, or Top-K score separation is probably insufficient. A numerically “healthy-looking” score distribution can still produce the wrong selection.

I think the most useful next experiment is a shadow-precision differential check on the indexer:

Keep the normal production FP8 path unchanged.
For selected layers/tokens, reconstruct the indexer K/Q inputs in BF16/FP32.
Compute a reference Top-K selection alongside the production FP8 selection.
Measure:
Top-K overlap / Jaccard similarity
first layer where selection diverges materially
divergence versus context length
tool vs no-tool prompts
GLM-5.1 versus DeepSeek-V3.2
Correlate the first selection divergence with the first hidden-state/logit divergence.

Something like:

layer FP8/BF16 top-k overlap hidden-state delta output

---

0-18 ~99% low coherent
19 97% low coherent
20 61% increases ...
21 34% high corrupt

That would give a much stronger discriminator than checking whether the indexer scores merely have reasonable variance.

If overlap remains healthy all the way through the failing request, the indexer can be largely cleared and instrumentation should move downstream. If overlap collapses specifically on GLM/SM90 as context/tool structure changes, we have a much tighter target: the FP8 K-cache / MQA selection path rather than the general FP8 GEMMs.

One additional thing I’d measure is first divergence, rather than only final corruption. MemForge’s general approach is to locate where measured behavior first departs from the reference before proposing a transformation; otherwise it’s very easy to optimize or patch a downstream symptom.

We’d be happy to prototype the shadow-precision/selection-overlap probe and contribute the instrumentation upstream if that would help. It can be kept env-gated so there is zero cost in normal serving.

MemForge: https://github.com/elmehdi-eljair/memforge

[WeiHaocheng]

Looks cool, let me take a look.