Fix: per-device staged-SO naming to stop paired-die AICPU device fault (507899 cascade)

[ChaoWao]

Symptom

st-onboard-a2a3 fails intermittently (~19% of runs after #870, vs ~10% before)
with a whole-suite collapse: the L2 tensormap_and_ringbuffer phase reports ~10
failed + ~23 errors at once —
simpler_init failed with code 507899, prepare_callable failed -1,
run_prepared failed with code 507018/507046/507901, rtMalloc failed: 507899.

It looks like an out-of-memory. It is not — npu-smi info taken right after
the failure shows every chip Health=OK with HBM ~3 GB / 64 GB used.

Root cause

507899 is [driver error:internal error] and 507901 is [hdc disconnect] —
a cascade after a device fault, not the cause. Surfacing the CANN device slog
(ASCEND_SLOG_PRINT_TO_STDOUT=1) shows the earliest error is an AICPU exception:

ProcessStarsAicpuErrorInfo: error from device(chipId:N, dieId:0/1),
                            an exception occurred during AICPU execution
PrintAicpuErrorInfo: Aicpu kernel execute failed,
                     soName=simpler_inner_<fp>.so, funcName=simpler_aicpu_exec,
                     errorCode=0x2a            # 507018 ACL_ERROR_RT_AICPU_EXCEPTION

simpler_aicpu_exec faults the whole chip; afterwards every rtStreamCreate /
rtMalloc on that chip returns 507899/507901, so the next test's simpler_init
fails and the suite collapses.

Two facts pin the mechanism:

1. On a2a3 the npu-smi Phy-IDs pair as die0/die1 of one Ascend910 (devices
   8/9, 4/5, …); the exception fires on both dies of one chip at the same
   instant — a chip-shared resource was corrupted.
2. The runtime stages SOs under the shared preinstall dir
   /usr/lib64/aicpu_kernels/0/aicpu_kernels_device/. The Feat: AICPU dispatcher bootstrap + cached AICore rtRegisterAllKernel handle (re-apply #537) #870 dispatcher wrote
   the AICPU runtime SO there under a content-fingerprint-only name
   simpler_inner_<fp>.so — identical across both dies. Paired dies share that
   filesystem, so both wrote/renamed/executed the same file; concurrent bootstrap
   corrupted the mmap'd image and trapped simpler_aicpu_exec.

A single-die 50× solo loop of the unregister/re-prepare/dedup tests never
reproduced; only the parallel multi-die suite did — consistent with a cross-die
shared-file race, not an intra-process use-after-free.

Fix

Make every SO staged under the shared preinstall dir per-device:

• Dispatcher inner SO → simpler_inner_<fp>_<device_id>.so. The real
  device_id (was hardcoded 0) is threaded from DeviceRunner through
  BootstrapDispatcher into both the host JSON reader name (MakeInnerSoBasename)
  and the device-side writer (DeviceArgs.device_id → MakeInnerSoPath). The
  process-level bootstrap cache is keyed by (fp, device_id).
• AICPU executor orchestration SO → libdevice_orch_<pid>_<cid>_<device_id>.so.
  device_id reaches the AICPU via a new trailing KernelArgs.device_id field,
  pushed in kernel.cpp via set_orch_device_id() and read by the executor via
  get_orch_device_id() (platform_regs). This SO was already pid-named (mostly
  safe); the suffix is defense-in-depth for colliding device-side pids.

Applied symmetrically to a2a3 and a5 (onboard + sim signatures in lockstep).

Testing

• tests/ut/cpp test_orch_so_file updated for the new signature + a new
  DistinctDeviceIdsProduceDistinctPaths case; full no-hardware ctest passes.
• Validated with 5/5 consecutive st-onboard-a2a3 passes on the inner-SO fix.
• Investigation + diagnosis recipe written up in
  docs/troubleshooting/a2a3-507899-aicpu-shared-so-fault.md.

🤖 Generated with Claude Code

[gemini-code-assist]

Note

Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds Ascend device-side logging environment variables to the st-onboard-a2a3 CI job, enabling slog output (including rtMalloc and OOM diagnostics) to be captured in CI logs for diagnostic visibility.

Changes

Ascend logging environment setup

Layer / File(s)	Summary
Ascend slog environment variables .github/workflows/ci.yml	Environment variables ASCEND_SLOG_PRINT_TO_STDOUT and ASCEND_GLOBAL_LOG_LEVEL are added to the st-onboard-a2a3 CI job to surface device-side slog output in CI logs.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Poem

A rabbit hops through CI logs with glee,
"Now slog shall flow for all to see!"
With Ascend's light and malloc's song,
Device diagnostics right along. 🐰📋

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The PR title describes fixing per-device SO naming to address a device fault, but the actual changeset only adds Ascend logging environment variables for diagnostic purposes in CI.	Update the title to reflect that this is a diagnostic-only change adding CANN logging to CI (e.g., 'DEBUG: dump CANN device slog to stdout on st-onboard-a2a3'), and explicitly note it should not be merged.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The description is related to the changeset, explaining the root cause investigation and diagnostic goals that motivate adding logging to the CI job.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

.github/workflows/ci.yml (1)

483-489: Consider adding a workflow guard to prevent accidental merge.

Since this is a diagnostic branch explicitly marked "do not merge", consider adding a step that fails if running on the main branch to provide an extra safeguard against accidental merge. However, given this is a temporary diagnostic branch and the PR review process should catch this, this is optional.

Optional: Example workflow guard

    steps:
      - name: Block merge to main (DEBUG branch only)
        if: github.ref == 'refs/heads/main'
        run: |
          echo "::error::This is a DEBUG branch and must not be merged to main"
          exit 1
      
      - name: Checkout repository
        uses: actions/checkout@v5

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/ci.yml around lines 483 - 489, This workflow adds
debug-only environment vars (ASCEND_SLOG_PRINT_TO_STDOUT and
ASCEND_GLOBAL_LOG_LEVEL) but lacks a guard to prevent accidental merges to main;
add a CI step near the top of the job that checks github.ref ==
'refs/heads/main' and exits non‑zero with an error message if true so the job
fails on main, ensuring the diagnostic branch cannot run on main; reference the
env keys ASCEND_SLOG_PRINT_TO_STDOUT and ASCEND_GLOBAL_LOG_LEVEL in the step’s
message to make intent explicit.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/ci.yml:
- Around line 483-489: This workflow adds debug-only environment vars
(ASCEND_SLOG_PRINT_TO_STDOUT and ASCEND_GLOBAL_LOG_LEVEL) but lacks a guard to
prevent accidental merges to main; add a CI step near the top of the job that
checks github.ref == 'refs/heads/main' and exits non‑zero with an error message
if true so the job fails on main, ensuring the diagnostic branch cannot run on
main; reference the env keys ASCEND_SLOG_PRINT_TO_STDOUT and
ASCEND_GLOBAL_LOG_LEVEL in the step’s message to make intent explicit.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: b636ee88-6924-41b8-93ad-1c627e41f623

📥 Commits

Reviewing files that changed from the base of the PR and between 1de6a90 and 7d1863d.

📒 Files selected for processing (1)

• .github/workflows/ci.yml