Docs: record the qwen 406 s decomposition and the four wrong answers - #1606
Conversation
|
Warning Review limit reached
Next review available in: 14 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
4ccfa00 to
ca51238
Compare
`qwen3_14b_decode` was the one scene test excluded from the a2a3 sweep, with a `--ignore` and a step of its own. hw-native-sys#1601 removed the exclusion by capping torch threads around goldens; this records how that root cause was found, because none of the intuitive answers was right and the next person will reach for them in the same order. The measured decomposition of the 406 s the case held a device: compile 59 s, fixture 13 s, golden 359 s, device busy for tens of milliseconds. None of the golden's 359 s was arithmetic — its reference walks 3584 small slice operations per layer and torch sizes its intra-op pool from the core count, so a 320-core host paid 6.35 s per layer against 1.05 s at 4 threads. Four hypotheses were measured and killed first: that the 38 GiB fixture dominated (13 s — the 38 GiB is `torch.cat([w] * 40)` replication, only ~2 GB is generated), that compilation dominated (59 s, real but never the majority), that per-case worker startup dominated (a 4.1 s floor is real but 57 cases summing to 615 s cannot explain a 1482 s step), and that the 320-thread golden figure was a local artifact — half true and the most instructive, since summing the independently measured phases closes to 6% of the CI window, which only works if CI's golden is the slow one too. Also records the method, which was the hard part: per-step wall clock does not separate queueing from work, but the `npu-lock` timestamps in the job log do — that is how 52% of the job turned out to be waiting for dies. `task-submit` localises those messages, so the recipe matches their ASCII parts and reads the three line shapes off `)...`, `(pid=NNNN)` and neither, which keeps it working regardless of locale and inside the repo's English-only lint. And it records why golden caching, golden vectorisation, a nightly golden split and `skip_golden` were each considered and dropped, so those are not re-derived. Indexed in docs/investigations/README.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 participant
Why
qwen3_14b_decodewas the one scene test excluded from the a2a3 sweep, with a--ignoreand a step of its own. #1601 removed the exclusion by capping torch threads around goldens.This records how that root cause was found, because none of the intuitive answers was right and the next person will reach for them in the same order. #1604's assignee needs this data too; right now it lives only in that issue's body.
What it records
The decomposition of the 406 s the case held a device (ci run 30507320146, job 90761014912):
generate_inputs, the 38 GiB fixturecompute_golden, 40 layersNone of the golden's 359 s was arithmetic: its reference walks 3584 small slice operations per layer, and torch sizes its intra-op pool from the core count, so a 320-core host paid 6.35 s/layer against 1.05 s at 4.
The four wrong answers, each with the measurement that killed it — the 38 GiB fixture (13 s; the 38 GiB is
torch.cat([w] * 40)replication so only ~2 GB is generated), compilation (59 s, real but never the majority), per-case worker startup (a 4.1 s floor is real, but 57 cases summing to 615 s cannot explain a 1482 s step), and "the 320-thread figure is a local artifact" — half true, and the most instructive, since summing the independently measured phases closes to 6% of the CI window, which only works if CI's golden is the slow one too.The method, which was the hard part. Per-step wall clock does not separate queueing from work; the
npu-locktimestamps in the job log do. That is how 52% of the job turned out to be waiting for dies. Also: how to time the host phases with no device at all, and that the vendor CCE kernels need the test's_CANN_INCLUDE_DIRSor they fail on a missing header and drop silently out of any total.Four dropped fixes with reasons — golden caching (key must hash the reference's own source or it goes stale silently and then confirms a wrong device result), vectorising the gather, a nightly golden split (a workflow with no owner, and the only check that 40 layers compose correctly put on a 24-hour delay), and
skip_goldenin CI.Testing
markdownlint-cli2— 0 errors across all 18 files indocs/investigations/mkdocs build --strict— exit 0docs/investigations/README.mdperdiscipline.md§4 — an unlinked entry is invisibleDocs only.