Skip to content

feat(gemm_a8w8_blockscale_bpreshuffle): add GLM-5.2 tuned configs (gfx950)#4243

Open
Raiden-Makoto wants to merge 1 commit into
ROCm:mainfrom
Raiden-Makoto:glm52-blockscale-bpreshuffle-gfx950
Open

feat(gemm_a8w8_blockscale_bpreshuffle): add GLM-5.2 tuned configs (gfx950)#4243
Raiden-Makoto wants to merge 1 commit into
ROCm:mainfrom
Raiden-Makoto:glm52-blockscale-bpreshuffle-gfx950

Conversation

@Raiden-Makoto

@Raiden-Makoto Raiden-Makoto commented Jul 15, 2026

Copy link
Copy Markdown

Motivation

The a8w8_blockscale_bpreshuffle GEMM has per-model tuned configs under
aiter/configs/model_configs/ for dsv3, mm2.5, and qwen — but none for GLM-5.2.
GLM-5.2's two 128-aligned MLA dense projections on gfx950 — q_b_proj
(N=4096, K=2048) and o_proj (N=6144, K=4096) — therefore miss the tuned-config
lookup and fall back to the default solution (aiter logs
not found tuned config in ... will use default config).

Note aiter already ships a8w8_bpreshuffle_tuned_gemm_glm5.2.csv, but that is the
per-tensor bpreshuffle path (schema has q_dtype_w). These projections use the
block-scale (128x128) path (a8w8_blockscale_bpreshuffle), which has no GLM-5.2
config. This PR adds one. Config coverage only — no code changes.

Technical Details

  • Adds one file:
    aiter/configs/model_configs/a8w8_blockscale_bpreshuffle_tuned_gemm_glm5.2.csv
    • 120 tuned rows for gfx950 / cu_num=256 covering both shapes
      (q_b_proj 4096,2048 and o_proj 6144,4096).
    • M coverage: prefill token counts (1024–65536) and decode / cuda-graph batch
      sizes (1–512).
    • Schema matches the base a8w8_blockscale_bpreshuffle_tuned_gemm.csv:
      gfx,cu_num,M,N,K,libtype,kernelId,splitK,us,kernelName,tflops,bw,errRatio
      (libtype ∈ {asm, ck, cktile}).
  • No code changes. AiterConfigs.get_config_file(...) globs
    model_configs/*a8w8_blockscale_bpreshuffle_tuned_gemm*.csv and
    update_config_files(...) merges + dedups (lowest us per shape) with the base
    at runtime, so the new file is picked up automatically.
  • Rows produced by the standard tuner
    (csrc/ck_gemm_a8w8_blockscale/gemm_a8w8_blockscale_tune.py --preshuffle), no
    manual edits.

Test Plan

  • Run the tuner for both shapes across the M range on gfx950 (MI355X, cu256).
  • Load GLM-5.2 and confirm the two shapes no longer emit the default-config
    fallback warning (i.e. get_config_file merges the new model_config).
  • op_tests/test_gemm_a8w8_blockscale.py on the added shapes.

Test Result

  • Tuner completed with 0 failures across all M for both shapes (gfx950, cu256).
  • With the file merged, the not found tuned config … will use default warnings
    for 4096,2048 and 6144,4096 no longer appear at load; both resolve to their
    tuned entries.

Submission Checklist

…x950)

The a8w8_blockscale_bpreshuffle GEMM has per-model tuned configs for
dsv3/mm2.5/qwen but none for GLM-5.2. Its 128-aligned MLA projections on
gfx950 -- q_b_proj (4096,2048) and o_proj (6144,4096) -- miss the tuned
lookup and fall back to the default solution. Add a model_configs file
with tuner-generated entries (gfx950/cu256, prefill + decode M). Auto-merged
by get_config_file's model_configs glob; no code changes.
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4243 --add-label <label>

@Raiden-Makoto
Raiden-Makoto marked this pull request as ready for review July 15, 2026 01:22
@Raiden-Makoto
Raiden-Makoto requested a review from a team July 15, 2026 01:22
@zufayu
zufayu requested a review from yifehuan July 15, 2026 02:20
@@ -0,0 +1,121 @@
gfx,cu_num,M,N,K,libtype,kernelId,splitK,us,kernelName,tflops,bw,errRatio
gfx950,256,1024,4096,2048,asm,5,1,18.4361,_ZN5aiter43fp8gemm_bf16_blockscale_BpreShuffle_128x128E,931.86,1023.77,0.0

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are some duplicated cases from other model config csv (same a8w8 blockscale) make ci pipeline failed, please recheck

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll take a look.

@Raiden-Makoto

Raiden-Makoto commented Jul 16, 2026

Copy link
Copy Markdown
Author

Fixed in f3130ae — dropped the shapes that duplicated qwen3.5_397b, merge is dedup-clean now. Could you please rerun CI?

Oseltamivir added a commit to SemiAnalysisAI/InferenceX that referenced this pull request Jul 18, 2026
The cookbook-verified v0.5.13.post1-rocm720-mi35x-20260618 build segfaults
in AITER gemm_a8w8_blockscale_bpreshuffle mid-profiling under the agentic
corpus (KV usage 0.95+, 5.3M pending tokens, untuned GLM-5.2 GEMM shapes
falling back to default configs - gfx950 tuned configs are not upstream
yet, ROCm/aiter#4243). Move to the newest rocm720 build for the latest
AITER fallback fixes.

Cap the conc list at 32 (was 64): the B300 sibling frontier showed
no-offload points beyond ~c48 are KV-cliff-dominated on the 131k-token
mean corpus, and MI355X prefill is slower still - c48/c64 only added
crash surface and sweep time for dominated points.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants