feat(glm53): 인덱서 fp32 head-gate split-K (glm53_indexer_gate_splitk, opt-in, EXP-9) - #227
Merged
Merged
Conversation
…te_splitk, opt-in) Indexer.forward 의 torch.mm(hidden.float(), _wp_fp32) ([M,4096]x[4096,16] fp32, 층당 1회 x 11) 을 cuBLAS 가 2블록 gemmSN 커널로 답한다: 유휴 GB10 47 us, 트레이스 (CUPTI) 86 us. (행, K-슬라이스 512) 마다 프로그램 하나 + fp32 atomic 의 split-K Triton 커널이 같은 곱을 10 us 에 낸다. M<=16(디코드) 만 라우팅, 나머지는 stock. - overlay/modules/glm53_indexer_gate_splitk: attention.py 오버레이(플릿 preimage a0870c31..., fi618 과 동일) + 커널 파일. 노브 VLLM_GLM53_INDEXER_GATE_SPLITK, 정확히 "1" 만 켬, 기본 0 = stock 과 같은 torch.mm 호출. - glm53_prefill_fastpath.py 의 융합 인덱서(FUSED_K_GATE=1)도 같은 헬퍼를 탄다. - 수치: fp32 누적 순서만 다름 — 300회/2,480행 max|diff| 2.4e-6, top-1 뒤집힘 0, top-4 집합 변화 0 (probes/indexer_gate_check.py). bit-exact 아님 -> 품질 브래킷. - 천장 11 x 40 us = 0.44 ms/스텝, C=1 의 ~0.65% (<1%): 단독 부팅 금지, EXP-7/8 부팅에 얹어 잰다. RUNBOOK EXP-9, MEASUREMENTS 원장. - tests: test_overlay_symbol_contracts 를 프로필 단위로 (dsv4 mla_indexer 와 glm53 glm53_tail_slot_persistent 가 같은 경로를 각자 소유 — 전역 "마지막 매니페스트" 맵이 glm53 attention.py 를 dsv4 indexer 에 대조하고 있었다). 새 계약 테스트. 41,165 checks OK. build/glm53 스냅샷 재구성 (33 overlays / 22 modules). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟡 Changes recommended
The split-K applicability predicate lacks CUDA/contiguity guards (risking incorrect behavior or crashes in non-CUDA/non-contiguous cases) and the kernel docstrings contain inconsistent measurement claims versus the PR’s documentation.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds an opt-in optimization for GLM-5.3’s sparse indexer fp32 head-gate GEMM on small decode shapes (M≤16) by routing through a split-K Triton kernel, while keeping the stock torch.mm path as the default and for larger shapes. The PR also updates wiring so both the standard and fused-indexer arms share the same head-gate helper, and extends tests/docs/probes to validate the new overlay and its numerics caveat.
Changes:
- Introduces
glm53_indexer_gate_splitkoverlay with a split-K Triton head-gate implementation and opt-in env knob (VLLM_GLM53_INDEXER_GATE_SPLITK=1). - Routes head-gate computation through a shared helper in both
attention.pyoverlay andglm53_prefill_fastpath.py, with stock fallback when the module isn’t mounted. - Adds per-profile overlay symbol contract checking, new contracts test, probe script, and updates runbook/measurements/kernel map.
File summaries
| File | Description |
|---|---|
tests/test_logic.py |
Makes overlay symbol contract checks profile-aware and adds glm53_indexer_gate_splitk contract test coverage. |
STEP_KERNEL_MAP.md |
Documents the new EXP-9 module and split-K head-gate optimization reference. |
RUNBOOK_KERNEL_CAMPAIGN2.md |
Adds EXP-9 runbook entry describing arming, numerics caveat, and measurement expectations. |
profiles/glm53.env |
Mounts the new module and declares default-off knob VLLM_GLM53_INDEXER_GATE_SPLITK=0. |
probes/indexer_gate_check.py |
Adds an offline numerics + timing probe (CUDA graph replay) for the split-K head gate. |
overlay/modules/glm53_model_wiring/glm53_prefill_fastpath.py |
Lazily imports and uses the shared head-gate helper with a stock fallback for non-mounted environments. |
overlay/modules/glm53_indexer_gate_splitk/requires |
Declares dependency on glm53_model_wiring. |
overlay/modules/glm53_indexer_gate_splitk/README.md |
Documents purpose, arming method, numerics caveat, and offline timing results. |
overlay/modules/glm53_indexer_gate_splitk/manifest.tsv |
Pins attention.py overlay and adds the new kernel helper module file. |
overlay/modules/glm53_indexer_gate_splitk/glm5next_attention.py |
Replaces the fp32 head-gate torch.mm call with the shared helper import/call. |
overlay/modules/glm53_indexer_gate_splitk/glm53_indexer_gate.py |
Implements split-K Triton kernel + helper routing and env-gated applicability. |
MEASUREMENTS.md |
Records offline measurements and rationale for routing only M≤16 through split-K. |
build/glm53/manifest.tsv |
Adds built artifacts for the pinned attention overlay and new kernel helper file. |
build/glm53/glm5next_attention.py |
Snapshot of the overlayed attention module with helper-based head-gate call. |
build/glm53/glm53_prefill_fastpath.py |
Snapshot of the model wiring helper import + callsite update. |
build/glm53/glm53_indexer_gate.py |
Snapshot of the split-K head-gate helper module. |
Review details
- Files reviewed: 16/16 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+72
to
+74
| def splitk_applicable(x: torch.Tensor, w: torch.Tensor) -> bool: | ||
| return (x.shape[0] <= MAX_M and w.dtype == torch.float32 and w.is_contiguous() | ||
| and w.shape[1] <= 16 and w.shape[0] % (_SPLIT * _BLOCK_K) == 0) |
Comment on lines
+12
to
+15
| product in 7 us. Both accumulate in fp32; only the summation order differs | ||
| (measured max |diff| 3e-5 on values of magnitude ~64, i.e. ~5e-7 relative, | ||
| 0 top-1 rank flips over the offline trials) -- not bit-exact, so this stays | ||
| an opt-in behind a numerics bracket. |
Comment on lines
+12
to
+15
| product in 7 us. Both accumulate in fp32; only the summation order differs | ||
| (measured max |diff| 3e-5 on values of magnitude ~64, i.e. ~5e-7 relative, | ||
| 0 top-1 rank flips over the offline trials) -- not bit-exact, so this stays | ||
| an opt-in behind a numerics bracket. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#226 (feat/glm53-prep-fused-v2) 위에 쌓인 브랜치 — #226 머지 후 base 를 main 으로 바꾸거나 함께 머지.
무엇
Indexer.forward의 fp32 head-gatetorch.mm(hidden.float(), _wp_fp32)([M,4096]×[4096,16], 층당 1회 × 11) 을 cuBLAS 가 2블록gemmSN커널로 답한다 (48 SM 중 2개): 유휴 GB10 47 us, 9월 1일 트레이스(CUPTI) 86 us. (행, K-슬라이스 512) 마다 프로그램 하나를 띄우고 fp32 atomic 으로 모으는 split-K Triton 커널이 같은 곱을 10 us 에 낸다. M<=16(디코드) 만 이 경로, 프리필·C>=3 은 stock 그대로.torch.mmtorch.mm유지(GB10, CUDA 그래프 리플레이,
probes/indexer_gate_check.py)수치 — bit-exact 아님
양쪽 다 fp32 누적, 합산 순서만 다르다. 300회/2,480행: max|diff| 2.4e-6 절대, 행 최대 대비 6.7e-7, top-1 뒤집힘 0, top-4 집합 변화 0. bf16 게이트가 순위를 뒤집는 오차(1e-2)보다 네 자릿수 아래지만 품질 브래킷(9/9, 한국어 0/16) 대상.
천장과 게이트
11 × ~40 us ≈ 0.44 ms/스텝 = C=1 66 ms 스텝의 ~0.65% — 원장 1% 규칙에 걸려 단독 부팅 금지. EXP-7/EXP-8 과 독립이라 그 부팅에 얹어 같이 잰다 (RUNBOOK EXP-9).
VLLM_GLM53_INDEXER_GATE_SPLITK=1 bash launchers/start-glm53-nvfp4-tp4.sh # 프로필 선언 키: caller env켜진 증거는 트레이스에서
gemmSN11개 →_gate_splitk_kernel11개 (부팅 로그 줄 없음 — 그래프 안에서 층마다 호출). 기본 0 = stock 과 같은torch.mm호출. 롤백 env 한 줄.파일
overlay/modules/glm53_indexer_gate_splitk/—attention.py오버레이 (플릿 preimagea0870c31…, fi618 동일; 변경은 import 1줄 + 호출 1줄) +glm53_indexer_gate.py(커널·헬퍼) + manifest/requires/READMEglm53_model_wiring/glm53_prefill_fastpath.py—VLLM_GLM53_FUSED_K_GATE=1팔의 융합 인덱서도 같은 헬퍼 (모듈 미탑재 시 stock 으로 폴백)profiles/glm53.env— MODULES 추가,VLLM_GLM53_INDEXER_GATE_SPLITK=0probes/indexer_gate_check.py— 수치(순위 뒤집힘 포함) + 그래프 리플레이 타이밍tests/test_logic.py— 계약 테스트;test_overlay_symbol_contracts를 프로필 단위로 (dsv4mla_indexer와 glm53glm53_tail_slot_persistent가mla/indexer.py를 각자 소유하는데 전역 "마지막 매니페스트" 맵이 glm53 의 attention.py 를 dsv4 indexer 에 대조해 거짓 실패). 41,165 checks OKbuild/glm53스냅샷 (33/22)🤖 Generated with Claude Code
Note
Medium Risk
Changes indexer head-gate numerics and sparse top-k inputs (opt-in only), so incorrect routing or ranking shifts could affect attention quality despite strong offline checks.
Overview
Adds
glm53_indexer_gate_splitk, an opt-in overlay that replaces the sparse indexer’s fp32 head-gatetorch.mm(hidden.float(), _wp_fp32)with a split-K Triton kernel whenVLLM_GLM53_INDEXER_GATE_SPLITK=1(profile default 0 = stock).Why: On decode shapes (M≤16), cuBLAS picks a two-block
gemmSNpath (~47–86 µs/layer × 11 layers). The new kernel does the same [M,4096]×[4096,16] multiply via per-(row, K-slice) programs and fp32 atomics (~10 µs offline). M>16 (prefill, larger verify batches) still usestorch.mm.Wiring:
attention.pyoverlay callshead_gate;glm53_prefill_fastpath.pyuses the same helper for the fused-K-gate arm (lazy import with stock fallback). Manifest pinsattention.pyand addsglm53_indexer_gate.py.Numerics: fp32 accumulation with different summation order — not bit-exact (max |diff| ~2.4e-6 offline, 0 top-1/top-4 ranking flips in probe trials). RUNBOOK EXP-9 quality bracket required; estimated ceiling ~0.65%/step (ride on EXP-7/8 boots, no solo boot).
Also:
probes/indexer_gate_check.py, MEASUREMENTS/RUNBOOK/STEP_KERNEL_MAP updates,test_glm53_indexer_gate_splitk_contracts, and per-profile overlay symbol contract tests (fixes false failures when dsv4 and glm53 both own differentmla/indexer.pyoverlays).Reviewed by Cursor Bugbot for commit 83d9eae. Bugbot is set up for automated code reviews on this repo. Configure here.