GLM-5.1-MXFP4 on AMD MI355X — massive GSM8K accuracy degradation on v0.5.12-rocm720-mi35x (off: 0.32, EAGLE-MTP: 0.18)

[functionstackx]

human

hi @HaiShaw @chunfangamd
+viz @Oseltamivir

mi355 glm5.1 fp4 is failing basic evals

Variant	gsm8k strict-match	gsm8k flexible-extract
--attention-backend nsa (no spec)	0.3177	0.3177
--attention-backend nsa + --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4	0.1774	0.1782

• InferenceX PR (OFF variant): [Handoff to @Oseltamivir Claude /loop] Update glm5.1-fp4-mi355x-sglang SGLang ROCm image to v0.5.12-rocm720-mi35x-20260517 SemiAnalysisAI/InferenceX#1441
  • Failing sweep run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26046393497
• InferenceX PR (MTP variant): [Handoff to @Oseltamivir Claude /loop] [Klaud Cold] Add glm5.1-fp4-mi355x-sglang-mtp recipe SemiAnalysisAI/InferenceX#1494
  • Failing sweep run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26018162860

AI Genearted

GLM-5.1-MXFP4 on AMD MI355X — severe GSM8K accuracy degradation on v0.5.12-rocm720-mi35x (off: 0.32, EAGLE-MTP: 0.18)

🤖 AI Generated — drafted by AI on behalf of @functionstackx via Claude Code /loop while triaging stuck PRs in SemiAnalysisAI/InferenceX.

Summary

Running amd/GLM-5.1-MXFP4 on AMD MI355X (gfx950) under lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260517 produces severely degraded GSM8K accuracy on lm-eval-harness:

Variant	gsm8k strict-match	gsm8k flexible-extract
--attention-backend nsa (no spec)	0.3177	0.3177
--attention-backend nsa + --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4	0.1774	0.1782

An instruction-tuned model of this class is expected to land at ~0.80+ on GSM8K. Our internal gate is 0.85 (which we'd consider lowering once the actual achievable number is known).

The server starts cleanly, the cuda graphs capture, and lm-eval runs to completion (~46 min) — there is no crash, no OOM, no kernel error. The output text is just much weaker on chain-of-thought math than expected for a 397B MoE instruction-tuned model.

Environment

• Image: lmsysorg/sglang-rocm:v0.5.12-rocm720-mi35x-20260517
• Model: amd/GLM-5.1-MXFP4 (MXFP4 quantization, 397B MoE)
• Hardware: 2× AMD MI355X (gfx950), tp=2
• lm-eval-harness: gsm8k (5-shot, generate_until, max_tokens=5376, temperature=0)
• InferenceX recipes affected: glm5.1-fp4-mi355x-sglang and glm5.1-fp4-mi355x-sglang-mtp

Reproduction

A. OFF variant — gsm8k = 0.3177

Launch server:

python3 -m sglang.launch_server --model-path amd/GLM-5.1-MXFP4 --host=0.0.0.0 --port 8891 \
  --tensor-parallel-size 2 --trust-remote-code \
  --cuda-graph-max-bs 256 --context-length 9472 --mem-fraction-static 0.85 \
  --tool-call-parser glm47 --reasoning-parser glm45 \
  --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 8}' \
  --nsa-prefill-backend tilelang --nsa-decode-backend tilelang \
  --kv-cache-dtype fp8_e4m3 \
  --tokenizer-worker-num 4 --disable-radix-cache

Run lm-eval:

python3 -m lm_eval --model local-chat-completions --apply_chat_template \
  --tasks gsm8k --output_path /tmp/eval_out --log_samples \
  --model_args 'model=amd/GLM-5.1-MXFP4,base_url=http://0.0.0.0:8891/v1/chat/completions,api_key=EMPTY,eos_string=</s>,max_retries=5,num_concurrent=64,timeout=1800,tokenized_requests=False,max_length=9472' \
  --gen_kwargs max_tokens=5376,temperature=0,top_p=1

Result:

|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.3177|± |0.0128|
|     |  |strict-match    | 5|exact_match|↑ |0.3177|± |0.0128|

B. EAGLE-MTP variant — gsm8k = 0.1774

Same launch as (A), but with these added flags:

  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

Result:

|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.1782|± |0.0105|
|     |  |strict-match    | 5|exact_match|↑ |0.1774|± |0.0105|

EAGLE-MTP halves accuracy vs the OFF variant on the same model, image, and prompts — temperature is 0, so this is a deterministic regression.

Links

• InferenceX PR (OFF variant): [Handoff to @Oseltamivir Claude /loop] Update glm5.1-fp4-mi355x-sglang SGLang ROCm image to v0.5.12-rocm720-mi35x-20260517 SemiAnalysisAI/InferenceX#1441
  • Failing sweep run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26046393497
• InferenceX PR (MTP variant): [Handoff to @Oseltamivir Claude /loop] [Klaud Cold] Add glm5.1-fp4-mi355x-sglang-mtp recipe SemiAnalysisAI/InferenceX#1494
  • Failing sweep run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/26018162860
• Tracking handoff: [AI Generated] [Handoff] out of 70+ image updates, 13 stuck Klaud Cold PRs need upstream coordination / scope decisions / [AI Generated] [Handoff] 70 多个镜像更新中 13 个 Klaud Cold PR 卡住，需上游协调或范围决策 SemiAnalysisAI/InferenceX#1511 (§10 + §12)

[chunfangamd]

@1am9trash ^^^

[Jacob0226]

We have traced the regression back to the Aiter library.
We are currently investigating which specific kernel is causing the issue.

[functionstackx]

thanks @Jacob0226

any chance that AITER & AMD SGLang could have regression testing & evals testing to do in depth testing of numeric accuracy of kernels before updating AITER/before releasing new models? @HaiShaw

[1am9trash]

It seems the GLM5.1 mxfp4 tp2 accuracy issue has been fixed. Ready to close.
cc @Jacob0226, @HaiShaw, @andyluo7

100%|██████████████████████████████████████████████████████████████████████| 1314/1314 [02:30<00:00,  8.73it/s]
Total latency: 150.632 s
Score: 0.943
Output throughput: 914.360 token/s

[functionstackx]

thank you @1am9trash

[HaiShaw]

thanks @Jacob0226

any chance that AITER & AMD SGLang could have regression testing & evals testing to do in depth testing of numeric accuracy of kernels before updating AITER/before releasing new models? @HaiShaw

We have AITER scout testing with SGLang in place, seems to be the case of missing TC.
cc @bingxche @sogalin @michaelzhang-ai please extend AITER testing with all Models and all dtypes supported, so as nightly tests.