perf(glm53): 게이트 이중 런치 제거 (−42/step) + 수용률 격차 실험 예약등록 - #96
Conversation
트레이스에서 게이트 커널이 topk의 정확히 2배(84 vs 42/step, 20us 간격,
동일 grid [18,1,1], 전부 graph id=0). 원인은 moe_runner.py:863 — 러너가
게이트를 직접 보유하고("apply it after the stream sync, so it can run
overlapped" — shared-expert aux stream 과 겹치는 자리) 넘겨받은
router_logits 를 무조건 덮어써 재계산한다. 우리 forward 의 명시적
self.gate(hidden) 호출은 그래서 100% 낭비였고, 실제로 쓰이는 건 항상
러너의 두 번째 결과였다.
수정: 우리 호출을 없애고 None 을 넘긴다. 효과:
- 스텝당 게이트 런치 84 -> 42 (−42, 전체의 −1.9%)
- 남은 한 번은 aux stream 과 겹쳐 도는 러너의 배치 — 이전보다 나은 자리
- 수치 불변 (같은 커널·같은 입력·어차피 러너 결과가 이겨 왔음)
dsv4 의 86/step 은 2포워드(46게이트×2)라 정상 — 이 버그는 glm53 전용.
같이: 수용률 격차 실험 예약등록 (63.1% vs dsv4 78.5% = ~1.66배
토큰/스텝). 헤드 노브가 둘로 분리돼 있음을 확인 — SPEC(드래프터 후보)과
TARGET(verify 로짓)이 각각 fp8. dsv4 가 양쪽 다 fp8 인 채 78.5%를
냈으므로 정밀도는 '기각할 가설'로 실험 설계: 부트 A/B 로 각 헤드만
끄고 pos-1 델타 ±2%p 판정 기준을 프로필에 문서화.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_5cfaba35-5550-4645-9aca-0f61819c9239) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d4983acf20
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # kernels exactly 2x the topk kernels, 84 vs 42/step, both eager, | ||
| # 20us apart, identical grids). Hand it None and let the runner's | ||
| # overlapped call be the only one. | ||
| router_logits = None |
There was a problem hiding this comment.
On the configured GPU/B12x path, self.experts forwards this value into the registered b12x_moe_forward_shared custom op, whose router_logits schema requires a torch.Tensor (probes/moe_prof.py:201-208,267-273). Argument validation occurs before _forward_impl can overwrite the placeholder with self.gate(hidden_states), so None causes the first MoE invocation to fail instead of removing the duplicate launch. Pass a tensor placeholder such as hidden_states, as the existing internal-router path does in overlay/modules/dsv4_model/nvidia_model.py:801-806.
Useful? React with 👍 / 👎.
|
추가 커밋: selector GEMV bf16 노브 + KERNEL_CAMPAIGN.md 재개정 지도 챕터 (같은 브랜치) |
Summary
트레이스 이상(게이트 84 = topk 42의 정확히 2배, 동일 grid 20µs 간격, 전부 이글)의 원인을 추적:
moe_runner.py:863에서 러너가 게이트를 직접 재계산하며 넘겨받은 logits을 덮어씀. 우리 forward의 명시적 호출이 100% 낭비였고, 러너의 호출은 shared-expert aux stream과 겹치도록 배치된 더 나은 자리.Changes
Glm5NextMoE.forward: 명시적self.gate(hidden)제거 → −42 런치/step (−1.9%), 수치 불변(어차피 러너 결과가 사용됨)profiles/glm53.env에 문서화게이트+topk 진짜 융합은 왜 안 했나
단일 CTA 설계(전체 N=288을 레지스터/스멤에 올려 topk까지)는 가중치 읽기 2.4MB를 1 CTA가 읽어야 해 게이트 GEMM 자체가 ~2배 느려짐 — 커널 1개를 아끼려 GEMM을 늦추는 역거래. µs/커널 상수가 부팅에서 크게 나오면 재검토.
🤖 Generated with Claude Code
Note
Low Risk
Performance-only dedup on a path whose outputs were already taken from the MoE runner; env changes are documentation for planned boots, not runtime logic.
Overview
MoE forward no longer calls
self.gatebeforeself.experts. It passesrouter_logits=Noneso only the fused MoE runner’s gate run remains—the one scheduled to overlap the shared-expert aux stream. Traces had shown every MoE layer launching the router twice per step (~42 redundant gate kernels/step); behavior is unchanged numerically because the runner was already overwriting precomputed logits.profiles/glm53.envadds a pre-registered ladder to test whether fp8 spec/target LM heads explain pos-1 acceptance (63.1% vs dsv4’s 78.5%): boot A withVLLM_SPEC_FP8_LM_HEAD=0, boot B withVLLM_TARGET_LM_HEAD_FP8=0, and retire the precision hypothesis if both deltas stay within ±2 pct.Reviewed by Cursor Bugbot for commit d4983ac. Bugbot is set up for automated code reviews on this repo. Configure here.