[Bug] Dynamic Register/Unregister Instability In A2A3 Sim CI

[ccyywwen]

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer

Description

#839 introduced dynamic post-init callable register/unregister coverage under:

tests/st/a2a3/tensormap_and_ringbuffer/dynamic_register/test_dynamic_register.py

The dynamic register/unregister path is still unstable in #861 CI. Two PR861
CI runs exposed failures in the same dynamic-register ST file on the same CI
job family:

1. PR861 CI #2723

   • Run: https://github.com/hw-native-sys/simpler/actions/runs/26559663566
   • Job: https://github.com/hw-native-sys/simpler/actions/runs/26559663566/job/78239340891
   • Job name: st-sim-a2a3 (ubuntu-latest, 3.10)
   • Environment: Ubuntu hosted runner, Python 3.10.20, pytest 9.0.3
   • Failing test:
     test_register_after_init_parallel_broadcast
   • Failure signature: rc=-11, Python segmentation fault in
     simpler/worker.py, line 2369, during worker.run(...).

2. PR861 CI #2749

   • Run: https://github.com/hw-native-sys/simpler/actions/runs/26575577396
   • Job: https://github.com/hw-native-sys/simpler/actions/runs/26575577396/job/78293886852
   • Job name: st-sim-a2a3 (ubuntu-latest, 3.10)
   • Environment: Ubuntu hosted runner, Python 3.10.20, pytest 9.0.3
   • Failing test:
     test_register_unregister_register_runs_each_time
   • Failure signature: pytest session timeout after 600s; scheduler reported
     the standalone test as hung for 490.1s; process exited with code 124.

These are not the same exact failure signature: CI #2723 failed with a
segmentation fault in the two-device parallel dynamic-register case, while CI
#2749 passed that case and instead hung in the single-device
unregister/re-register reuse case. They should still be tracked together as a
PR839 dynamic register/unregister stability issue because both failures occur in
the same feature area and the same ST file.

Steps to Reproduce

Run PR861 CI on the host-device_mapped-region branch with the standard CI
workflow and inspect the Ubuntu A2A3 simulation ST job:

CI / st-sim-a2a3 (ubuntu-latest, 3.10)

The relevant full CI invocations were:

PR861 CI #2723:
https://github.com/hw-native-sys/simpler/actions/runs/26559663566

PR861 CI #2749:
https://github.com/hw-native-sys/simpler/actions/runs/26575577396

For local focused reproduction, run the dynamic-register ST cases on A2A3 sim:

pytest tests/st/a2a3/tensormap_and_ringbuffer/dynamic_register/test_dynamic_register.py \
  --platform a2a3sim --device 0-1 -p no:xdist --pto-session-timeout 600

The two observed failing standalone cases can also be targeted directly:

pytest tests/st/a2a3/tensormap_and_ringbuffer/dynamic_register/test_dynamic_register.py::test_register_after_init_parallel_broadcast \
  --platform a2a3sim --device 0-1 -p no:xdist --pto-session-timeout 600

pytest tests/st/a2a3/tensormap_and_ringbuffer/dynamic_register/test_dynamic_register.py::test_register_unregister_register_runs_each_time \
  --platform a2a3sim --device 0 -p no:xdist --pto-session-timeout 600

Because the failures appear intermittent, a single local run may pass. Looping
these focused cases is likely needed to reproduce the instability.

Expected Behavior

Dynamic post-init register and unregister should be deterministic and safe in
A2A3 simulation:

• test_register_after_init_parallel_broadcast should successfully broadcast a
  post-init CTRL_REGISTER to both chip children, return only after each child
  has prepared the callable, and then run the dynamically registered cid on
  both chips without crashing.
• test_register_unregister_register_runs_each_time should successfully run a
  dynamically registered cid, unregister it, reuse the freed cid slot on a
  subsequent register, and run the re-registered callable without hanging.
• The full st-sim-a2a3 (ubuntu-latest, 3.10) CI job should complete without
  segfaults, hangs, or session-level timeouts.

Actual Behavior

Observed in PR861 CI #2723:

[scheduler] START standalone test_register_after_init_parallel_broadcast
(rt=tensormap_and_ringbuffer, dev=2) pid=8668 devices=[6, 10]

standalone test_register_after_init_parallel_broadcast
(rt=tensormap_and_ringbuffer, dev=2) [FAIL rc=-11 57.8s, devices=[6, 10]]

tests/st/a2a3/tensormap_and_ringbuffer/dynamic_register/
test_dynamic_register.py Fatal Python error: Segmentation fault

File ".../site-packages/simpler/worker.py", line 2369 in run
File ".../dynamic_register/test_dynamic_register.py", line 234
in test_register_after_init_parallel_broadcast

Process completed with exit code 1.

Observed in PR861 CI #2749:

[scheduler] START standalone test_register_after_init_parallel_broadcast
(rt=tensormap_and_ringbuffer, dev=2) pid=8857 devices=[6, 10]

standalone test_register_after_init_parallel_broadcast
(rt=tensormap_and_ringbuffer, dev=2) [PASS 21.4s, devices=[6, 10]]

[scheduler] START standalone test_register_unregister_register_runs_each_time
(rt=tensormap_and_ringbuffer, dev=1) pid=9389 devices=[8]

[pytest] TIMEOUT: session exceeded 600s (10min) limit

HUNG standalone test_register_unregister_register_runs_each_time
(rt=tensormap_and_ringbuffer, dev=1) pid=9389 devices=[8]
elapsed=490.1s descendants=[9565, 9566]

Process completed with exit code 124.

This indicates that the PR839 dynamic register/unregister path can fail in at
least two ways under CI load: a post-register worker.run(...) segfault in the
two-device broadcast case, and a hang in the unregister/re-register cid reuse
case.

Git Commit ID

825f0fd

CANN Version

No response

Driver Version

No response

Host Platform

Linux (x86_64)

Additional Context

No response

[ChaoWao]

Re: the hang (CI #2749, rc=124) — it's a sim oversubscription livelock, not a register/unregister logic bug

Root-caused the test_register_unregister_register_runs_each_time hang. It is not specific to the dynamic register/unregister feature — it's the sim init handshake livelocking under CPU oversubscription. Register/unregister just makes it more likely because each worker.run() re-runs the AICPU↔AICore startup handshake.

Mechanism. On sim every AICPU and AICore is a host thread. The startup handshake is a pair of mutual busy-spins with no yield:

• AICPU — SchedulerContext::handshake_all_cores: while (hank->aicore_regs_ready == 0) {} / while (hank->aicore_done == 0) {}
• AICore — aicore_execute: while (my_hank->aicpu_ready == 0) {} / while (my_hank->aicpu_regs_ready == 0) {}

On st-sim-a2a3 (ubuntu-latest) the runner has 4 vCPUs and --max-parallel auto resolves to 4, so 4 cases run concurrently, each forking chip subprocesses that each spawn ~block_dim AICore threads. The non-yielding spins then monopolize the few cores and starve the very threads they wait on → the handshake never completes → 490 s hang / session timeout.

A gdb snapshot of a wedged chip subprocess (8× domain_rank_map on --cpus=2) confirmed it: the AICPU thread parked in SchedulerContext::handshake_all_cores while ~100 AICore host threads busy-spun in aicore_execute.

Fix. #893 adds SPIN_WAIT_HINT() to the four init-handshake spins (((void)0) on onboard, sched_yield() on sim) and widens the sim deinit-ACK timeout (1 s → 10 s). #894 follows up with docstrings + a corrected vCPU note.

Verified in a --cpus=2 container (worse than CI's 4 vCPU): dynamic_register at --max-parallel 4 → 8/8 rounds pass (timed out before the fix); domain_rank_map 8 concurrent × 8 rounds → 64/64 (was 0/48). See docs/troubleshooting/sim-oversubscription-hang.md.

Repro note. The isolated --device 0-1 -p no:xdist recipe above usually won't reproduce it — a single case at low parallelism doesn't oversubscribe. The pressure on CI comes from other concurrent cases sharing the runner. To reproduce, constrain CPUs and run concurrently, e.g. docker run --cpus=2 ... looping several instances, or the full suite at --max-parallel 4 on a 2-core box.

Note: this covers only the hang signature. The rc=-11 SIGSEGV in CI #2723 is a separate residual and is not addressed by #893 — it did not reproduce locally (extreme oversubscription only ever wedged into a hang). Tracking that separately.

[ChaoWao]

Resolved — root-caused and fixed (merged)

Both failure signatures in this issue traced to the sim CPU-oversubscription bug family (on sim every AICPU/AICore/host-orchestrator is a host thread competing for a few CPUs, which widens spin/timeout/lifetime race windows by orders of magnitude). Each was root-caused and fixed:

1. Hang — CI #2749 (rc=124, test_register_unregister_register_runs_each_time)
The AICPU↔AICore init handshake used bare busy-spins that don't yield, so under oversubscription they starved the threads they waited on → the handshake never completed. Fixed in #893 (SPIN_WAIT_HINT() in the handshake spins; also widened the sim AICore deinit timeout 1s→10s so OS-scheduler jitter doesn't false-trip it). Onboard unaffected — SPIN_WAIT_HINT() is a no-op on hardware.

2. Segfault — CI #2723 (rc=-11 in worker.run, test_register_after_init_parallel_broadcast)
Two distinct concurrency bugs on the worker.run path, both reproduced and AddressSanitizer-proven on sibling tests sharing the exact code:

• Host hierarchical drain use-after-free (fix: serialize drain() teardown against scheduler loop (host UAF) #901): Orchestrator::drain() → Ring::reset_to_empty() freed TaskSlotStates while the scheduler thread was still inside on_task_complete (the active-task count reaches 0 mid-completion). dynamic_register's register/unregister path exercises this orchestrator directly. ASAN reproduced it within ~8–40 runs; fixed by serializing drain()'s teardown against the scheduler loop via a loop_mutex.
• Device sync_start drain null-deref (fix: null-guard pending_task in sync_start drain election (sim segfault) #898): a null pending_task deref + non-atomic read in the AICPU tensormap_and_ringbuffer scheduler's handle_drain_mode, exposed under concurrent drains.

Verification

dynamic_register's two cases on current main (all fixes merged), at 8× concurrency under a CPU-constrained docker --cpus=2 container (harsher than CI's ~4 vCPU): 88/88 runs clean — zero rc=139 (segfault) and zero rc=124 (hang). Before the fixes, the underlying UAF/livelock reproduced within tens of runs.

All fixes are merged: #893 (hang + deinit), #898 (sync_start segfault), #901 (host drain UAF). Closing as resolved — please reopen if either signature recurs in st-sim-*.