feat(osar): AR phase 리포트를 랭크별로 — 편차 귀속 (레버 5 대체) - #297
Conversation
…s the AR skew attribution The one-shot AR is 5.4 ms of a 61 ms C=1 step: p10 22.6 us is the RDMA round trip, the ~17 us above it is arrival spread. Only rank 0 reported its phases, and a rank's own wait says nothing about who it waits for. Every rank now prints '[osar] phase rank=R ...'; the rank with the smallest wait is the last arriver, max-min is what skew work could recover. Logging only; same cadence, same deltas. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_7161c8da-6695-4a5a-9539-2ba577c17f73) |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟢 Approval recommended
The changes are limited to logging/commentary and the updated fprintf formatting/arguments are consistent, with no behavioral impact on the collective logic.
Pull request overview
This PR extends one-shot AllReduce (OSAR) phase telemetry so the periodic phase report is emitted by every rank (not only rank 0), enabling per-rank wait-time comparison to attribute arrival skew while keeping the reporting cadence and delta-based aggregation unchanged.
Changes:
- Emit
[osar] phase ...telemetry from all ranks by removing theg_rank == 0gating. - Prefix each phase report line with
rank=%dso logs can be grouped/compared per rank. - Expand the inline comment to document the skew-attribution interpretation of per-rank
wait.
File summaries
| File | Description |
|---|---|
| overlay/modules/tp_oneshot_ar/dsv4_oneshot_ar.cu | Changes OSAR proxy phase-report logging to print per-rank with a rank tag and updated rationale comment. |
| build/glm53/dsv4_oneshot_ar.cu | Mirrors the per-rank phase-report logging/tagging change in the glm53 build copy. |
| build/dsv4/dsv4_oneshot_ar.cu | Mirrors the per-rank phase-report logging/tagging change in the dsv4 build copy. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
AR 편차 귀속(레버 5 대체): one-shot AR 의 phase 리포트를 랭크 0 만이 아니라 네 랭크 모두
[osar] phase rank=R ...로 찍습니다(로깅만, 동일 주기·델타). 랭크의 wait 는 자기 도착 → 마지막 도착까지의 시간이라 wait 가 가장 작은 랭크가 남들을 기다리게 하는 랭크이고, max−min 이 편차 작업으로 회수할 수 있는 몫입니다(AR 5.4 ms/스텝 중 ~1.7 ms). 체인의 다음 부팅부터 데이터가 쌓이고~/glm53-logs/ar_skew.py로 집계합니다.🤖 Generated with Claude Code
Note
Low Risk
Observability-only: more stderr from all ranks; no change to AllReduce behavior or timing paths.
Overview
One-shot AR phase telemetry now prints from every rank, not only rank 0. The proxy still emits the same ~10s delta averages (guard, copy, wait, reduce, size-split wait), but each line is tagged
[osar] phase rank=R ...so you can compare per-rank wait and attribute arrival skew (smallest wait ≈ the rank others wait on).The change is applied identically in
dsv4_oneshot_ar.cuunderoverlay/modules/tp_oneshot_ar/and thebuild/dsv4/build/glm53copies. Comments were expanded to document the skew-attribution rationale; no change to kernels, RDMA, or collective logic.Reviewed by Cursor Bugbot for commit 6c21161. Bugbot is set up for automated code reviews on this repo. Configure here.