Skip to content

Add B200 config: glm5-fp4-sglang-mtp #1087

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
functionstackx merged 1 commit into from
Apr 18, 2026
Merged

Add B200 config: glm5-fp4-sglang-mtp #1087

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1983,6 +1983,26 @@ glm5-fp4-b200-sglang:
- { tp: 8, ep: 1, conc-start: 4, conc-end: 4 }
- { tp: 4, ep: 1, conc-start: 4, conc-end: 256 }

glm5-fp4-b200-sglang-mtp:
image: lmsysorg/sglang:v0.5.10.post1-cu130
model: nvidia/GLM-5-NVFP4
model-prefix: glm5
runner: b200
precision: fp4
framework: sglang
multinode: false
seq-len-configs:
- isl: 1024
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-start: 4, conc-end: 4, spec-decoding: mtp }
- { tp: 4, ep: 1, conc-start: 4, conc-end: 256, spec-decoding: mtp }
- isl: 8192
osl: 1024
search-space:
- { tp: 8, ep: 1, conc-start: 4, conc-end: 4, spec-decoding: mtp }
- { tp: 4, ep: 1, conc-start: 4, conc-end: 256, spec-decoding: mtp }

# NOTE: At the time of submission, https://cookbook.sglang.io/autoregressive/GLM/GLM-5
# does not have a B300-specific recipe, so this config reuses the existing
# GLM-5 FP4 B200 SGLang recipe as-is until B300-specific tuning is available.
Expand Down
91 changes: 91 additions & 0 deletions benchmarks/single_node/glm5_fp4_b200_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
#!/usr/bin/env bash

source "$(dirname "$0")/../benchmark_lib.sh"

check_env_vars \
MODEL \
TP \
CONC \
ISL \
OSL \
RANDOM_RANGE_RATIO \
RESULT_FILENAME

if [[ -n "$SLURM_JOB_ID" ]]; then
echo "JOB $SLURM_JOB_ID running on $SLURMD_NODENAME"
fi

nvidia-smi

hf download "$MODEL"

pip install --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

export SGL_ENABLE_JIT_DEEPGEMM=1
export SGLANG_ENABLE_SPEC_V2=1

SERVER_LOG=/workspace/server.log
PORT=${PORT:-8888}


echo "CONC: $CONC, ISL: $ISL, OSL: $OSL"

EVAL_CONTEXT_ARGS=""
if [ "${EVAL_ONLY}" = "true" ]; then
setup_eval_context
EVAL_CONTEXT_ARGS="--context-length $EVAL_MAX_MODEL_LEN"
fi
# Start GPU monitoring (power, temperature, clocks every second)
start_gpu_monitor

set -x
PYTHONNOUSERSITE=1 python3 -m sglang.launch_server --model-path=$MODEL --host=0.0.0.0 --port=$PORT \
--trust-remote-code \
--tensor-parallel-size=$TP \
--data-parallel-size 1 --expert-parallel-size 1 \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--kv-cache-dtype fp8_e4m3 --quantization fp8 \
Copy link
Copy Markdown
Contributor

@claude claude Bot Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new script passes --quantization fp8 and --kv-cache-dtype fp8_e4m3 (copied verbatim from the FP8 sibling script), but the model is nvidia/GLM-5-NVFP4 which uses pre-quantized NVFP4 weights. The correct flag for NVFP4 weights is --quantization modelopt_fp4, as used by the existing benchmarks/single_node/glm5_fp4_b200.sh (line 42). With the wrong quantization flag, SGLang will either fail at model load or produce invalid benchmark results.

Extended reasoning...

What the bug is and how it manifests

The new launch script benchmarks/single_node/glm5_fp4_b200_mtp.sh (line 48) passes --kv-cache-dtype fp8_e4m3 --quantization fp8 to sglang.launch_server. However, the model being loaded is nvidia/GLM-5-NVFP4 — a model whose weights are already pre-quantized in NVIDIA's NVFP4 (modelopt_fp4) format. The fp8 quantization flags are appropriate for FP8-quantized models like zai-org/GLM-5-FP8, not for NVFP4 weights.

The specific code path that triggers it

When a sweep job runs for the glm5-fp4-b200-sglang-mtp config, the harness selects glm5_fp4_b200_mtp.sh as the launch script. That script calls:

python3 -m sglang.launch_server --model-path= ... --kv-cache-dtype fp8_e4m3 --quantization fp8

with MODEL=nvidia/GLM-5-NVFP4. SGLang will attempt to apply FP8 quantization to weights that are already stored in NVFP4 format.

Why existing safeguards do not catch this

This is a semantic error invisible to bash -n syntax checking (which the PR author confirms passed). The YAML config and model field correctly reference nvidia/GLM-5-NVFP4 with precision: fp4, but the quantization directive in the launch script contradicts this at runtime. There is no static analysis that cross-validates the quantization flag against the model weights format.

Impact

SGLang will either: (a) reject the conflicting quantization scheme and fail to start the server, causing the sweep job to error out; or (b) silently misinterpret the NVFP4 weights under an FP8 quantization scheme, resulting in benchmark numbers that do not represent actual NVFP4 performance. Either outcome invalidates any measurements collected under this config. The config is already labeled sweep-enabled, meaning it could be swept as-is.

How to fix it

Replace line 48 of benchmarks/single_node/glm5_fp4_b200_mtp.sh:

# Before (wrong):
--kv-cache-dtype fp8_e4m3 --quantization fp8 \
# After (correct):
--quantization modelopt_fp4 \

This matches the existing non-MTP counterpart benchmarks/single_node/glm5_fp4_b200.sh line 42. Whether to also add --kv-cache-dtype fp8_e4m3 for FP8 KV cache should be verified against the FP4 B200 recipe.

Step-by-step proof

  1. The YAML config glm5-fp4-b200-sglang-mtp sets model: nvidia/GLM-5-NVFP4 and precision: fp4.
  2. A sweep job instantiates glm5_fp4_b200_mtp.sh with MODEL=nvidia/GLM-5-NVFP4.
  3. Line 48 of that script passes --quantization fp8 to sglang.launch_server.
  4. SGLang sees NVFP4 weights but is told to use FP8 quantization — a mismatch.
  5. Compare with benchmarks/single_node/glm5_fp4_b200.sh line 42, which correctly uses --quantization modelopt_fp4 for the same nvidia/GLM-5-NVFP4 model.
  6. The PR description itself flags this as an unresolved TODO: 'Decide whether fp8 launch flags are correct for NVFP4 weights, or if script should switch to fp4-specific quantization.'

--attention-backend nsa \
--nsa-decode-backend trtllm --nsa-prefill-backend trtllm \
--moe-runner-backend flashinfer_trtllm \
--cuda-graph-max-bs $CONC --max-running-requests $CONC \
--mem-fraction-static 0.85 \
--chunked-prefill-size 32768 --max-prefill-tokens 32768 \
--enable-flashinfer-allreduce-fusion --disable-radix-cache \
--stream-interval 30 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--model-loader-extra-config '{"enable_multithread_load": true}' $EVAL_CONTEXT_ARGS > $SERVER_LOG 2>&1 &

SERVER_PID=$!

# Wait for server to be ready
wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"

pip install -q datasets pandas

run_benchmark_serving \
--model "$MODEL" \
--port "$PORT" \
--backend vllm \
--input-len "$ISL" \
--output-len "$OSL" \
--random-range-ratio "$RANDOM_RANGE_RATIO" \
--num-prompts "$((CONC * 10))" \
--max-concurrency "$CONC" \
--result-filename "$RESULT_FILENAME" \
--result-dir /workspace/ \
--use-chat-template

# After throughput, run evaluation only if RUN_EVAL is true
if [ "${RUN_EVAL}" = "true" ]; then
run_eval --framework lm-eval --port "$PORT"
append_lm_eval_summary
fi

# Stop GPU monitoring
stop_gpu_monitor
set +x
10 changes: 10 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1596,3 +1596,13 @@
- "Mirrors the qwen3.5-fp4-b200-sglang non-MTP recipe and adds EAGLE speculative decoding (num-steps=3, eagle-topk=1, num-draft-tokens=4)"
- "Configs: 1k1k and 8k1k, TP=4/EP=1 conc 4-128 with spec-decoding=mtp"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX

- config-keys:
- glm5-fp4-b200-sglang-mtp
description:
- "Add GLM-5 NVFP4 B200 SGLang MTP benchmark (draft)"
- "Image: lmsysorg/sglang:v0.5.10.post1-cu130"
- "Model: nvidia/GLM-5-NVFP4"
- "Follows the glm5-fp8-b200-sglang launch recipe as requested, plus EAGLE speculative decoding (num-steps=3, eagle-topk=1, num-draft-tokens=4) behind SGLANG_ENABLE_SPEC_V2=1"
- "Configs: 1k1k and 8k1k, TP8/EP1 conc 4-4 + TP4/EP1 conc 4-256 with spec-decoding=mtp"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX
Loading