Fix: cap torch threads for goldens, and put qwen back in the scene-test sweep by ChaoWao · Pull Request #1601 · hw-native-sys/simpler

ChaoWao · 2026-07-30T12:22:00Z

Replaces this PR's earlier nightly-split approach. Measuring the case showed the golden's cost was thread-pool overhead, not work — capping threads removes it, so there is nothing left to move to a nightly and the numeric guard stays on every PR.

What was wrong

qwen3_14b_decode held a device for 406 s (ci run 30507320146, job 90761014912) while the device itself was busy for tens of milliseconds. It was excluded from the general a2a3 sweep with a --ignore and given a step and an 1800 s budget of its own, because at that size it would have eaten the sweep's 600 s session budget — the budget that makes the sweep a hang detector.

Decomposed on this repo's a2a3 box, one device:

Phase	Wall
kernel compilation — 36 incores + 1 orchestration	59 s
`generate_inputs`, the 38 GiB fixture	13 s
`compute_golden`, 40 layers at 320 torch threads	359 s
	~431 s vs 406 s measured

The golden was 80–85% of it, and none of that was arithmetic.

Root cause

The reference implementation walks 3584 small slice operations per layer, and torch sizes its intra-op pool from the core count. On a 320-core host every one of those calls forks and joins hundreds of threads to move a few KiB:

threads	320	64	16	8	4	2	1
per layer	6.35 s	1.31 s	1.13 s	1.07 s	1.05 s	1.59 s	1.82 s

Five to eight times the cost of the work, flat from 4 to 16, turning back up below 4.

Nothing in simpler_setup/, conftest.py or the workflows had ever set a thread count, so every golden paid this on every many-core host. All three under simpler_setup/goldens/ are built from per-tile Python loops — qwen is merely the densest (8 for-range loops against 2 each).

Change

The cap goes where all three goldens pass — around the two compute_golden call sites in scene_test.py — rather than into any one golden. It only ever lowers the limit (a caller who asked for fewer keeps theirs) and restores the previous value on the way out, exception included.

That takes qwen's golden to ~38–43 s and the whole case to ~115 s: an ordinary member of a sweep whose own wall is ~505 s across four dies. So the --ignore, the dedicated step and its 1800 s budget are all deleted — the case is nothing special now and runs like every other one.

Testing

Results unaffected — same fixture golden-computed at 320 and at 8 threads is bit-identical across out, k_cache and v_cache (torch.equal true, max|diff| = 0). Checked rather than assumed, since thread count can reorder float reductions.
Cap verified end to end: lowers 320 → 8 inside, restores 320 after, leaves a caller's 2 alone, and restores after an exception raised inside the block.
Effect measured through the helper: 0.95 s per layer → ~38 s for 40 layers, against 6.35 s / 254 s uncapped.
Compile figures measured per source with KernelCompiler — 35 ordinary incores at 1.1–1.3 s each, orchestration 6.6 s, and the vendor FAI kernel as the outlier at 8.4 s / 1330 KiB for its AIV variant.
yaml.safe_load on ci.yml; ast.parse on scene_test.py; markdownlint-cli2 clean
The sweep has not yet run with qwen in it. That is the check that matters on this PR: st-onboard-a2a3 should stay well inside its 600 s budget with the case included.

Docs

.claude/skills/testing/SKILL.md loses the HEAVY_IGNORE quarantine row and the command carrying it, plus three ci.yml:<line> references this change would have invalidated. The README's cost section is rewritten around the measurements — it reported "~5 min wall, ~49 s golden" from an idle die, which is the 4–16-thread figure and does not describe CI.

coderabbitai · 2026-07-30T12:28:14Z

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 979e8d8c-c522-4742-b24c-ccddcfd70008

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

📝 Walkthrough

Walkthrough

The Qwen3-14B decode test now separates functional PR validation from full golden comparison. CI runs the functional case, while a new scheduled or manually triggered nightly workflow runs numeric validation on self-hosted a2a3 hardware. Documentation records the execution split and measured costs.

Changes

Qwen3-14B golden gating

Layer / File(s)	Summary
Separate functional and golden cases `examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/test_qwen3_14b_decode.py`, `examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/README.md`	The PR case sets `skip_golden=True`; a matching `manual=True` case performs the full golden comparison. Documentation records the two cases and measured timing costs.
PR and nightly workflow execution `.github/workflows/ci.yml`, `.github/workflows/nightly.yml`, `.claude/skills/testing/SKILL.md`	CI labels and runs the functional step, while `nightly.yml` schedules or manually runs the golden case with architecture-specific device allocation and timeout settings.
Documented validation strategy `docs/ci.md`	The CI guide describes nightly numeric guards, separate scheduling, golden replay costs, and monitoring expectations.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PullRequest
  participant CI
  participant Nightly
  participant Qwen3Decode
  PullRequest->>CI: trigger functional validation
  CI->>Qwen3Decode: run skip_golden case
  Nightly->>Qwen3Decode: run --manual only golden case
  Qwen3Decode-->>Nightly: return numeric comparison result

Possibly related PRs

hw-native-sys/simpler#1470: Adds the per-case skip_golden hooks used by this gating approach.

Poem

I’m a rabbit hopping through the gate,
Functional checks run while golden waits.
Nightly brings the reference bright,
Forty layers tested under moonlight.
/_/\
( o.o )

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The title describes capping torch threads and restoring qwen to the sweep, but the PR actually splits the golden into a nightly workflow and updates docs.	Rewrite the title to mention the qwen3_14b_decode PR/nightly split and the skip_golden/manual golden workflow.
Description check	⚠️ Warning	The description argues for keeping qwen in PR by capping threads, but the PR actually moves the numeric golden to nightly and updates CI/docs.	Replace it with a summary of the workflow split: PR runs skip_golden, nightly runs the manual golden, plus the documentation updates.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 3

🧹 Nitpick comments (1)

.github/workflows/nightly.yml (1)
25-35: 🩺 Stability & Availability | 🔵 Trivial

Give the nightly numeric gate an explicit failure route.

Add a failure notification/issue step, or document an owned alerting path. Otherwise this sole 40-layer numeric guard can fail silently despite the documentation requiring active monitoring.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/nightly.yml around lines 25 - 35, Update the
qwen-golden-a2a3 nightly job to provide an explicit failure route: add a
notification or issue-creation step that runs when the job fails, or document an
owned alerting path within the workflow. Ensure failures of this sole 40-layer
numeric guard cannot remain unreported.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yml:
- Around line 627-629: Measure golden-path overhead as an estimate rather than
an exact real cost: in .github/workflows/ci.yml lines 627-629, remove the
reversed subtraction or explicitly label it approximate; in
examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/README.md lines 201-203,
use the correct subtraction direction and avoid calling it the “real cost”; and
in .github/workflows/nightly.yml lines 62-66, describe the comparison as an
estimate unless adding direct compute_golden timing.

In `@docs/ci.md`:
- Around line 61-64: Update the design-principle statement in docs/ci.md to
qualify that --platform is the only hardware/platform filter, or explicitly
limit the claim to the PR matrix; keep the documented --manual only nightly path
and its behavior unchanged.

In `@examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/README.md`:
- Around line 174-177: Update the “StressBatch16Seq3500” entry in the README
table to qualify “every PR” as every relevant PR, specifically when a2a3_changed
and st_affected select the st-onboard-a2a3 scene-test job; leave the nightly
golden-case description unchanged.

---

Nitpick comments:
In @.github/workflows/nightly.yml:
- Around line 25-35: Update the qwen-golden-a2a3 nightly job to provide an
explicit failure route: add a notification or issue-creation step that runs when
the job fails, or document an owned alerting path within the workflow. Ensure
failures of this sole 40-layer numeric guard cannot remain unreported.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: dfae4394-969e-42b3-b713-748de054b3d1

📥 Commits

Reviewing files that changed from the base of the PR and between 72148a2 and 75a2a4e.

📒 Files selected for processing (6)

.claude/skills/testing/SKILL.md
.github/workflows/ci.yml
.github/workflows/nightly.yml
docs/ci.md
examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/README.md
examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/test_qwen3_14b_decode.py

…st sweep `qwen3_14b_decode` held a device for 406 s (ci run 30507320146, job 90761014912) while the device itself was busy for tens of milliseconds. It was excluded from the general a2a3 sweep with a `--ignore` and given a step and a 1800 s budget of its own, because at that size it would have eaten the sweep's 600 s session budget — the budget that makes the sweep a hang detector. Decomposed on this repo's a2a3 box, one device: kernel compilation, 36 incores + 1 orchestration 59 s generate_inputs, the 38 GiB fixture 13 s compute_golden, 40 layers at 320 torch threads 359 s ------ ~431 s vs 406 s measured The golden was 80-85% of it, and none of that was arithmetic. Its reference implementation walks 3584 small slice operations per layer, and torch sizes its intra-op pool from the core count, so on a 320-core host every one of those calls forks and joins hundreds of threads to move a few KiB: threads 320 64 16 8 4 2 1 per layer 6.35s 1.31s 1.13s 1.07s 1.05s 1.59s 1.82s Five to eight times the cost of the work, flat from 4 to 16, turning back up below 4. Nothing in `simpler_setup/`, `conftest.py` or the workflows had ever set a thread count, so every golden paid this on every many-core host — all three under `simpler_setup/goldens/` are built from per-tile Python loops, qwen is merely the densest. So the cap goes where all of them pass, around the two `compute_golden` call sites in `scene_test.py`, rather than into any one golden. It only ever lowers the limit and restores the previous value on the way out, exception included. Results are unaffected: the same fixture golden-computed at 320 and at 8 threads is bit-identical across `out`, `k_cache` and `v_cache`. That takes qwen's golden to ~38-43 s and the whole case to ~115 s, which is an ordinary member of a sweep whose own wall is ~505 s across four dies. The `--ignore`, the dedicated step and its 1800 s budget are all deleted; the case is nothing special now and runs like every other one. `.claude/skills/testing/SKILL.md` loses the `HEAVY_IGNORE` quarantine row and the command that carried it, plus three `ci.yml:<line>` references that this change would have invalidated anyway. The README's cost section is rewritten around the measurements; it had reported "~5 min wall, ~49 s golden" from an idle die, which is the 4-16-thread figure and does not describe CI. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`qwen3_14b_decode` was the one scene test excluded from the a2a3 sweep, with a `--ignore` and a step of its own. hw-native-sys#1601 removed the exclusion by capping torch threads around goldens; this records how that root cause was found, because none of the intuitive answers was right and the next person will reach for them in the same order. The measured decomposition of the 406 s the case held a device: compile 59 s, fixture 13 s, golden 359 s, device busy for tens of milliseconds. None of the golden's 359 s was arithmetic — its reference walks 3584 small slice operations per layer and torch sizes its intra-op pool from the core count, so a 320-core host paid 6.35 s per layer against 1.05 s at 4 threads. Four hypotheses were measured and killed first: that the 38 GiB fixture dominated (13 s — the 38 GiB is `torch.cat([w] * 40)` replication, only ~2 GB is generated), that compilation dominated (59 s, real but never the majority), that per-case worker startup dominated (a 4.1 s floor is real but 57 cases summing to 615 s cannot explain a 1482 s step), and that the 320-thread golden figure was a local artifact — half true and the most instructive, since summing the independently measured phases closes to 6% of the CI window, which only works if CI's golden is the slow one too. Also records the method, which was the hard part: per-step wall clock does not separate queueing from work, but the `npu-lock` timestamps in the job log do — that is how 52% of the job turned out to be waiting for dies. `task-submit` localises those messages, so the recipe matches their ASCII parts and reads the three line shapes off `)...`, `(pid=NNNN)` and neither, which keeps it working regardless of locale and inside the repo's English-only lint. And it records why golden caching, golden vectorisation, a nightly golden split and `skip_golden` were each considered and dropped, so those are not re-derived. Indexed in docs/investigations/README.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…1606) `qwen3_14b_decode` was the one scene test excluded from the a2a3 sweep, with a `--ignore` and a step of its own. #1601 removed the exclusion by capping torch threads around goldens; this records how that root cause was found, because none of the intuitive answers was right and the next person will reach for them in the same order. The measured decomposition of the 406 s the case held a device: compile 59 s, fixture 13 s, golden 359 s, device busy for tens of milliseconds. None of the golden's 359 s was arithmetic — its reference walks 3584 small slice operations per layer and torch sizes its intra-op pool from the core count, so a 320-core host paid 6.35 s per layer against 1.05 s at 4 threads. Four hypotheses were measured and killed first: that the 38 GiB fixture dominated (13 s — the 38 GiB is `torch.cat([w] * 40)` replication, only ~2 GB is generated), that compilation dominated (59 s, real but never the majority), that per-case worker startup dominated (a 4.1 s floor is real but 57 cases summing to 615 s cannot explain a 1482 s step), and that the 320-thread golden figure was a local artifact — half true and the most instructive, since summing the independently measured phases closes to 6% of the CI window, which only works if CI's golden is the slow one too. Also records the method, which was the hard part: per-step wall clock does not separate queueing from work, but the `npu-lock` timestamps in the job log do — that is how 52% of the job turned out to be waiting for dies. `task-submit` localises those messages, so the recipe matches their ASCII parts and reads the three line shapes off `)...`, `(pid=NNNN)` and neither, which keeps it working regardless of locale and inside the repo's English-only lint. And it records why golden caching, golden vectorisation, a nightly golden split and `skip_golden` were each considered and dropped, so those are not re-derived. Indexed in docs/investigations/README.md.

The SDMA quarantine was a pair of `--ignore=<path>` arguments. `pytest --ignore` on a path that no longer exists **exits 0 and says nothing** — measured — so moving either demo directory would have dropped the quarantine silently and landed both tests back beside the `aicore_op_timeout` fault-injection test. That collision is the whole hazard: provisioning the SDMA workspace creates 48 device-only STARS streams that sit in the device fault domain, so a later AICore fault on that device costs ~306 s instead of ~0.3 s (hw-native-sys#1425, contained in hw-native-sys#1406). `@pytest.mark.sdma` replaces the path matching, and carries the whole consequence rather than one arbitrary piece of it. One declaration now drives three things: - **The Worker is built with `enable_sdma=True`.** Both pytest construction sites in `conftest.py` and both standalone sites in `scene_test.py` read it, the latter from `cls.pytestmark` so `python test_x.py` behaves like the pytest path. - **The L2 Worker pool stops mixing capabilities.** The key gains the flag and reuse tests it, because an enable_sdma Worker holds its STARS streams for life: handing it to a test that did not ask for them would spread the teardown hazard to every later L2 case on that device. The existing same-device retire loop then performs the swap, so no new teardown path. - **SDMA sorts last.** `sort_key` gains a term, keyed off the marker rather than the class because the fault-injection tests are plain functions with no `_st_level`. Fault injection therefore always runs on a device that has never provisioned — which is what actually addresses the interaction, rather than merely quarantining it. With the capability expressible, `prefetch_async_demo` becomes an ordinary L2 `@scene_test` class. It was a hand-rolled Worker only because `CASES` had no channel to `Worker.__init__`, and it paid for that by forfeiting golden comparison, case parametrization, `--rounds`, `--case` and the dispatcher's device allocation. 160 lines become 89, and its verification — `torch.equal(out, src)` — is now a one-line `compute_golden`. The framework's default orchestration includes turned out to be sufficient, so no new passthrough was needed there. `sdma_async_completion_demo` keeps its hand-rolled L3 Worker for now and takes the marker for ordering and CI selection only; converting it means expressing a comm domain through `CASES`, which is a larger change and independent of this one. CI selects with `-m sdma` / `-m "not sdma"`, so the sweep and the dedicated step can no longer drift apart. **The dedicated step stays until hw-native-sys#1425 is fixed**: in-session ordering separates the cases, but a fault on a device that has already provisioned still costs minutes, so they must not share a device. Separately, `docs/capability-survey.md` carried four `ci.yml:<line>` citations, three already invalidated by this session's edits — `:607` lands on a blank line, `:884` on a `cmake --build`, `:628-643` on a dep_gen comment, after the qwen step was deleted in hw-native-sys#1601 and `detect-changes` reworked in hw-native-sys#1589 / hw-native-sys#1590 / hw-native-sys#1607. Line numbers into a file this active cannot be maintained, so all four become greppable anchors. `grep -rIn 'ci\.yml:[0-9]'` is now empty repo-wide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The SDMA quarantine was a pair of `--ignore=<path>` arguments. `pytest --ignore` on a path that no longer exists **exits 0 and says nothing** — measured — so moving either demo directory would have dropped the quarantine silently and landed both tests back beside the `aicore_op_timeout` fault-injection test. That collision is the whole hazard: provisioning the SDMA workspace creates 48 device-only STARS streams that sit in the device fault domain, so a later AICore fault on that device costs ~306 s instead of ~0.3 s (hw-native-sys#1425, contained in hw-native-sys#1406). `@pytest.mark.sdma` replaces the path matching, and carries the whole consequence rather than one arbitrary piece of it. One declaration now drives three things: - **The Worker is built with `enable_sdma=True`.** Both pytest construction sites in `conftest.py` and both standalone sites in `scene_test.py` read it, the latter from `cls.pytestmark` so `python test_x.py` behaves like the pytest path. - **The L2 Worker pool stops mixing capabilities.** The key gains the flag and reuse tests it, because an enable_sdma Worker holds its STARS streams for life: handing it to a test that did not ask for them would spread the teardown hazard to every later L2 case on that device. The existing same-device retire loop then performs the swap, so no new teardown path. - **SDMA sorts last.** `sort_key` gains a term, keyed off the marker rather than the class because the fault-injection tests are plain functions with no `_st_level`. This is what will make merging the dedicated CI step back into the sweep safe once hw-native-sys#1425 is fixed; today `-m` already separates them, so it matters for local full runs. With the capability expressible, `prefetch_async_demo` becomes an ordinary L2 `@scene_test` class. It was a hand-rolled Worker only because `CASES` had no channel to `Worker.__init__`, and it paid for that by forfeiting golden comparison, case parametrization, `--rounds`, `--case` and the dispatcher's device allocation. 160 lines become 89, and its verification — `torch.equal(out, src)` — is now a one-line `compute_golden`. The framework's default orchestration includes turned out to be sufficient, so no new passthrough was needed there. The conversion also moves it from the Resource phase to the L2 phase, which runs after it, so it provisions on a device the fault-injection cases have already finished with. `sdma_async_completion_demo` keeps its hand-rolled L3 Worker for now and takes the marker for ordering and CI selection only; converting it means expressing a comm domain through `CASES`, which is a larger change and independent of this one. CI selects with `-m sdma` / `-m "not sdma"`, so the sweep and the dedicated step can no longer drift apart. **The dedicated step stays until hw-native-sys#1425 is fixed**: the two phases run their jobs in parallel across devices, so only separate device pools keep an SDMA provisioning away from a fault injection. `.claude/skills/testing/SKILL.md` moves with the mechanism throughout — not only the mirror-CI command, but the two instructions that told readers to extract `--ignore` sets from `ci.yml` and to grep it when a test passes alone and fails in the sweep. Both now name the marker; following the skill reproduces CI rather than a mechanism that no longer exists. Separately, `docs/capability-survey.md` carried four `ci.yml:<line>` citations, three already invalidated by this session's edits — `:607` lands on a blank line, `:884` on a `cmake --build`, `:628-643` on a dep_gen comment, after the qwen step was deleted in hw-native-sys#1601 and `detect-changes` reworked in hw-native-sys#1589 / hw-native-sys#1590 / hw-native-sys#1607. Line numbers into a file this active cannot be maintained, so all four become greppable anchors. `grep -rIn 'ci\.yml:[0-9]'` is now empty repo-wide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The SDMA quarantine was a pair of `--ignore=<path>` arguments. `pytest --ignore` on a path that no longer exists **exits 0 and says nothing** — measured — so moving either demo directory would have dropped the quarantine silently and landed both tests back beside the `aicore_op_timeout` fault-injection test. That collision is the whole hazard: provisioning the SDMA workspace creates 48 device-only STARS streams that sit in the device fault domain, so a later AICore fault on that device costs ~306 s instead of ~0.3 s (hw-native-sys#1425, contained in hw-native-sys#1406). `@pytest.mark.sdma` replaces the path matching, and carries the whole consequence rather than one arbitrary piece of it. One declaration now drives three things: - **The Worker is built with `enable_sdma=True`.** Both pytest construction sites in `conftest.py` and both standalone sites in `scene_test.py` read it, the latter from `cls.pytestmark` so `python test_x.py` behaves like the pytest path. - **The L2 Worker pool stops mixing capabilities.** The key gains the flag and reuse tests it, because an enable_sdma Worker holds its STARS streams for life: handing it to a test that did not ask for them would spread the teardown hazard to every later L2 case on that device. The existing same-device retire loop then performs the swap, so no new teardown path. - **SDMA sorts last.** `sort_key` gains a term, keyed off the marker rather than the class because the fault-injection tests are plain functions with no `_st_level`. This is what will make merging the dedicated CI step back into the sweep safe once hw-native-sys#1425 is fixed; today `-m` already separates them, so it matters for local full runs. With the capability expressible, `prefetch_async_demo` becomes an ordinary L2 `@scene_test` class. It was a hand-rolled Worker only because `CASES` had no channel to `Worker.__init__`, and it paid for that by forfeiting golden comparison, case parametrization, `--rounds`, `--case` and the dispatcher's device allocation. 160 lines become 89, and its verification — `torch.equal(out, src)` — is now a one-line `compute_golden`. The framework's default orchestration includes turned out to be sufficient, so no new passthrough was needed there. The conversion also moves it from the Resource phase to the L2 phase, which runs after it, so it provisions on a device the fault-injection cases have already finished with. `sdma_async_completion_demo` keeps its hand-rolled L3 Worker for now and takes the marker for ordering and CI selection only; converting it means expressing a comm domain through `CASES`, which is a larger change and independent of this one. CI selects with `-m sdma` / `-m "not sdma"`, so the sweep and the dedicated step can no longer drift apart. **The dedicated step stays until hw-native-sys#1425 is fixed**: the two phases run their jobs in parallel across devices, so only separate device pools keep an SDMA provisioning away from a fault injection. `.claude/skills/testing/SKILL.md` moves with the mechanism throughout — not only the mirror-CI command, but the two instructions that told readers to extract `--ignore` sets from `ci.yml` and to grep it when a test passes alone and fails in the sweep. Both now name the marker; following the skill reproduces CI rather than a mechanism that no longer exists. Separately, `docs/capability-survey.md` carried four `ci.yml:<line>` citations, three already invalidated by this session's edits — `:607` lands on a blank line, `:884` on a `cmake --build`, `:628-643` on a dep_gen comment, after the qwen step was deleted in hw-native-sys#1601 and `detect-changes` reworked in hw-native-sys#1589 / hw-native-sys#1590 / hw-native-sys#1607. Line numbers into a file this active cannot be maintained, so all four become greppable anchors. `grep -rIn 'ci\.yml:[0-9]'` is now empty repo-wide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…#1609) The SDMA quarantine was a pair of `--ignore=<path>` arguments. `pytest --ignore` on a path that no longer exists **exits 0 and says nothing** — measured — so moving either demo directory would have dropped the quarantine silently and landed both tests back beside the `aicore_op_timeout` fault-injection test. That collision is the whole hazard: provisioning the SDMA workspace creates 48 device-only STARS streams that sit in the device fault domain, so a later AICore fault on that device costs ~306 s instead of ~0.3 s (#1425, contained in #1406). `@pytest.mark.sdma` replaces the path matching, and carries the whole consequence rather than one arbitrary piece of it. One declaration now drives three things: - **The Worker is built with `enable_sdma=True`.** Both pytest construction sites in `conftest.py` and both standalone sites in `scene_test.py` read it, the latter from `cls.pytestmark` so `python test_x.py` behaves like the pytest path. - **The L2 Worker pool stops mixing capabilities.** The key gains the flag and reuse tests it, because an enable_sdma Worker holds its STARS streams for life: handing it to a test that did not ask for them would spread the teardown hazard to every later L2 case on that device. The existing same-device retire loop then performs the swap, so no new teardown path. - **SDMA sorts last.** `sort_key` gains a term, keyed off the marker rather than the class because the fault-injection tests are plain functions with no `_st_level`. This is what will make merging the dedicated CI step back into the sweep safe once #1425 is fixed; today `-m` already separates them, so it matters for local full runs. With the capability expressible, `prefetch_async_demo` becomes an ordinary L2 `@scene_test` class. It was a hand-rolled Worker only because `CASES` had no channel to `Worker.__init__`, and it paid for that by forfeiting golden comparison, case parametrization, `--rounds`, `--case` and the dispatcher's device allocation. 160 lines become 89, and its verification — `torch.equal(out, src)` — is now a one-line `compute_golden`. The framework's default orchestration includes turned out to be sufficient, so no new passthrough was needed there. The conversion also moves it from the Resource phase to the L2 phase, which runs after it, so it provisions on a device the fault-injection cases have already finished with. `sdma_async_completion_demo` keeps its hand-rolled L3 Worker for now and takes the marker for ordering and CI selection only; converting it means expressing a comm domain through `CASES`, which is a larger change and independent of this one. CI selects with `-m sdma` / `-m "not sdma"`, so the sweep and the dedicated step can no longer drift apart. **The dedicated step stays until #1425 is fixed**: the two phases run their jobs in parallel across devices, so only separate device pools keep an SDMA provisioning away from a fault injection. `.claude/skills/testing/SKILL.md` moves with the mechanism throughout — not only the mirror-CI command, but the two instructions that told readers to extract `--ignore` sets from `ci.yml` and to grep it when a test passes alone and fails in the sweep. Both now name the marker; following the skill reproduces CI rather than a mechanism that no longer exists. Separately, `docs/capability-survey.md` carried four `ci.yml:<line>` citations, three already invalidated by this session's edits — `:607` lands on a blank line, `:884` on a `cmake --build`, `:628-643` on a dep_gen comment, after the qwen step was deleted in #1601 and `detect-changes` reworked in #1589 / #1590 / #1607. Line numbers into a file this active cannot be maintained, so all four become greppable anchors. `grep -rIn 'ci\.yml:[0-9]'` is now empty repo-wide.

test_a5_aicpu_topology_fallback compiles src/a5/platform/onboard/host/ aicpu_topology_probe.cpp, which now includes common/acl_hal_device.h, but the target never gained src/common/platform/include. ut, ut-a2a3 and ut-a5 all fail at "fatal error: common/acl_hal_device.h: No such file or directory". The a2a3 sibling target already carries the directory; add it to the a5 one. Also drop the "Qwen3-14B 40-layer decode (a2a3)" ci.yml step, which the rebase reintroduced. hw-native-sys#1601 removed it deliberately when qwen went back into the general scene-test sweep, so keeping it runs the 38 GiB fixture twice per a2a3 onboard job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

test_a5_aicpu_topology_fallback compiles src/a5/platform/onboard/host/ aicpu_topology_probe.cpp, which now includes common/acl_hal_device.h, but the target never gained src/common/platform/include. ut, ut-a2a3 and ut-a5 all fail at "fatal error: common/acl_hal_device.h: No such file or directory". The a2a3 sibling target already carries the directory; add it to the a5 one. Also drop the "Qwen3-14B 40-layer decode (a2a3)" ci.yml step, which the rebase reintroduced. hw-native-sys#1601 removed it deliberately when qwen went back into the general scene-test sweep, so keeping it runs the 38 GiB fixture twice per a2a3 onboard job.

The chip-init path makes direct driver calls that bypass ACL (halMemCtl / halGetDeviceInfo* on a2a3, halResMap / halGetDeviceInfo* / dsmi_get_device_info on a5), so they index the driver-visible space and must translate through acl_to_hal_device_id. Nothing exercised that translation: no CI job sets ASCEND_RT_VISIBLE_DEVICES, so every job ran the identity path and a call site that forgot to translate stayed green. tests/st/vis_isolation runs dummy_task in a subprocess with the granted card exposed as logical 0. init_aicore_register_addresses and probe_aicpu_topology run on every onboard bring-up, so one scene test covers every translated site on both arches. The subprocess is load-bearing: the variable must be set before ACL init, and mutating it in-process would leak into the session's pooled workers. Carrying this as a scene test rather than the two ci.yml steps keeps the selection in the test tree, so the sweep and the check cannot drift apart -- the direction hw-native-sys#1601 and hw-native-sys#1609 took for qwen and SDMA. The onboard jobs already run `pytest examples tests/st`, and the platforms marker keeps it off the sim runners, where a driver-level remap has nothing to verify.

coderabbitai Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Outdated

Comment thread docs/ci.md Outdated

Comment thread examples/a2a3/tensormap_and_ringbuffer/qwen3_14b_decode/README.md Outdated

ChaoWao force-pushed the ci/qwen-golden-to-nightly branch from 75a2a4e to 84efa02 Compare July 30, 2026 14:15

ChaoWao changed the title ~~Move the qwen 40-layer numeric check to a nightly workflow~~ Fix: cap torch threads for goldens, and put qwen back in the scene-test sweep Jul 30, 2026

ChaoWao merged commit 14f44a7 into hw-native-sys:main Jul 31, 2026
18 checks passed

ChaoWao deleted the ci/qwen-golden-to-nightly branch July 31, 2026 00:46

This was referenced Jul 31, 2026

[Code Health] Scene tests hold an NPU lock through their CPU phases, and recompile every kernel on every CI run #1604

Open

Docs: record the qwen 406 s decomposition and the four wrong answers #1606

Merged

ChaoWao mentioned this pull request Jul 31, 2026

Fix: make SDMA a declared capability, and drop stale ci.yml line refs #1609

Merged

7 tasks

ChaoWao mentioned this pull request Jul 31, 2026

Docs: realign ci.md with the gates it describes #1617

Merged

4 tasks

ChaoZheng109 mentioned this pull request Aug 1, 2026

Fix: honor ASCEND_RT_VISIBLE_DEVICES at direct-HAL device-id sites #1600

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix: cap torch threads for goldens, and put qwen back in the scene-test sweep - #1601

Fix: cap torch threads for goldens, and put qwen back in the scene-test sweep#1601
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:ci/qwen-golden-to-nightly

ChaoWao commented Jul 30, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 30, 2026 •

edited

Loading

Review skipped

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (2 warnings)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ChaoWao commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong

Root cause

Change

Testing

Docs

Uh oh!

coderabbitai Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (2 warnings)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ChaoWao commented Jul 30, 2026 •

edited

Loading

coderabbitai Bot commented Jul 30, 2026 •

edited

Loading