Refactor: isolate hierarchical worker run state

[Crane-Liu]

Summary

• add per-run identity, lifecycle state, completion counts, and first-error isolation to the hierarchical worker
• namespace TensorMap entries by run and serialize construction/consumption access
• remove completion dependence on the global drain/error slot while keeping Worker.run() blocking and returning None
• compact Ring storage only when the worker is globally quiescent

Series scope

This is PR 1 of the worker async-preparation series. It intentionally does not expose RunHandle, admit overlapping runs, or change device ordering. Those behaviors remain in later independently reviewable PRs.

Review follow-ups applied

• Keep run bookkeeping non-throwing on the scheduler and worker threads. on_task_failed_cb is invoked from Scheduler::worker_done, which runs on a WorkerThread whose on_complete_ call sits outside its try/catch, and on_consumed_cb runs inside Scheduler::run(), which has no try/catch. Both reach run accounting, so an unknown run id there would terminate the process. find_run() now resolves without throwing, and a count underflow is recorded as the run's error instead of thrown.
• Carry the orchestration failure message onto the run fence. _fail_run_submission takes an optional message, so a submission failure is retrievable from the run and not only from the raising caller — required once RunHandle lands in PR 2.
• Close the allocator-compaction race. begin_run() takes only runs_mu_, so testing quiescence under the scheduler loop mutex alone still allowed: test quiescent → release runs_mu_ → another thread begin_run() + allocate its first slot → reset_to_empty() invalidates it. compact_if_quiescent() now holds runs_mu_ across both the quiescence test and reset_to_empty(). This is sufficient because both allocator call sites (alloc(), reserve_outputs_and_slot()) are gated behind current_building_run(), so no slot can be allocated while no run is building. Lock order is scheduler loop mutex then runs_mu_, matching the scheduler's on_consumed path.
• Close the run inside the block that releases it, so a failure in close/fail_run_submission cannot strand the run id or skip run-owned resource cleanup.
• Document that no transition assigns RunPhase::PREPARED while one run builds at a time.

Added regression tests: error reporting and consumption for an already-released run are ignored rather than fatal, and a failed submission carries its message to the fence. The compaction race is fixed structurally (the test and the reset are now under one mutex); a deterministic unit test would need an injection hook to stop a thread mid-compaction, and a timing-based one would be flaky, so none was added.

Testing

• pre-commit: all hooks passed
• C++ unit tests: 60/60 passed (ctest --test-dir tests/ut/cpp/build -LE requires_hardware)
• Python unit tests: 804 passed, 2 skipped (pytest tests/ut)

Tested commit: 7b7b206d.

The architecture-probe and hardware runs quoted in earlier revisions were against 4d7eb574, which is no longer this branch's head, and have not been re-run. GitHub simulation CI remains the automated regression stage.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR replaces drain-based orchestration with explicit per-run lifecycle APIs, run-scoped task and error tracking, run-aware synchronized TensorMap operations, scheduler failure callbacks, and post-fence cleanup across C++, Python bindings, workers, tests, and documentation.

Changes

Run-fenced runtime

Layer / File(s)	Summary
Run lifecycle and orchestration state src/common/hierarchical/types.*, src/common/hierarchical/orchestrator.*, tests/ut/cpp/hierarchical/test_orchestrator.cpp	Runs now have IDs, phases, task counters, completion fences, failure state, and explicit begin/close/fail/wait/release operations.
Run-scoped TensorMap src/common/hierarchical/tensormap.*, tests/ut/cpp/hierarchical/test_tensormap.cpp	Tensor producers are keyed by (RunId, TensorKey), synchronized with a mutex, and erased per run.
Failure reporting and slot compaction src/common/hierarchical/scheduler.*, src/common/hierarchical/worker_manager.*, src/common/hierarchical/ring.*, tests/ut/cpp/hierarchical/test_scheduler.cpp	Scheduler failures attach to originating runs, WorkerManager error storage is removed, and ring reset is restricted to globally safe conditions.
Worker and Python lifecycle integration src/common/hierarchical/worker.cpp, python/bindings/worker_bind.h, python/simpler/*.py, docs/*, tests/ut/py/test_worker/*	Worker execution, bindings, deferred domain release, recursive coordination, and documentation now use run close/wait/release semantics.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Worker
  participant Orchestrator
  participant Scheduler
  participant CommDomain
  Worker->>Orchestrator: begin_run()
  Worker->>Orchestrator: close_run_submission(run_id)
  Scheduler->>Orchestrator: report_task_error(slot, message)
  Scheduler->>Orchestrator: on_consumed(task_slot)
  Worker->>Orchestrator: wait_run(run_id)
  Worker->>CommDomain: execute pending releases
  Worker->>Orchestrator: release_run(run_id)

Loading

Possibly related PRs

• hw-native-sys/simpler#1379: Adds pipelined request-session execution that coordinates with the orchestration run lifecycle.
• hw-native-sys/simpler#1436: Updates hierarchical submission and dependency inference paths touched by run-scoped bookkeeping.

Poem

A bunny waits where run fences gleam,
No drain can interrupt the dream.
Tensor keys hop by run and name,
Failures find their proper frame.
When tasks are done, the rings reset—
And carrots bloom after wait_run is met.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly matches the main change: isolating hierarchical worker run state.
Description check	✅ Passed	The description is detailed and directly describes the run-state, TensorMap, and quiescence changes in the pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/simpler/worker.py`:
- Around line 5681-5694: Update the run teardown flow surrounding
`_cleanup_l3_l2_regions()` and `_orch._release_run(run_id)` to preserve any
primary exception from the orchestration callback or `_wait_run()`. Capture
cleanup and release failures for reporting or exception chaining, but ensure the
original run failure is re-raised when one exists; only propagate teardown
failures when no primary exception occurred.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e9958a00-5747-4a22-ac63-7a65126cc0e3

📥 Commits

Reviewing files that changed from the base of the PR and between d0bc661 and 4d7eb57.

📒 Files selected for processing (28)

• docs/comm-domain.md
• docs/hierarchical_level_runtime.md
• docs/orchestrator.md
• docs/remote-l3-worker-design.md
• docs/scheduler.md
• docs/task-flow.md
• python/bindings/worker_bind.h
• python/simpler/orchestrator.py
• python/simpler/task_interface.py
• python/simpler/worker.py
• src/common/hierarchical/orchestrator.cpp
• src/common/hierarchical/orchestrator.h
• src/common/hierarchical/ring.cpp
• src/common/hierarchical/ring.h
• src/common/hierarchical/scheduler.cpp
• src/common/hierarchical/scheduler.h
• src/common/hierarchical/tensormap.cpp
• src/common/hierarchical/tensormap.h
• src/common/hierarchical/types.cpp
• src/common/hierarchical/types.h
• src/common/hierarchical/worker.cpp
• src/common/hierarchical/worker_manager.cpp
• src/common/hierarchical/worker_manager.h
• tests/ut/cpp/hierarchical/test_orchestrator.cpp
• tests/ut/cpp/hierarchical/test_scheduler.cpp
• tests/ut/cpp/hierarchical/test_tensormap.cpp
• tests/ut/py/test_worker/test_error_propagation.py
• tests/ut/py/test_worker/test_l3_l2_orch_comm.py

[Crane-Liu]

CI triage update: ut-a5 and st-onboard-a5 both failed before checkout or test execution. In each job, actions/checkout@v5 could not clean the runner workspace:

EACCES: permission denied, rmdir .../simpler/.pytest_cache/v/cache

Affected jobs:

• https://github.com/hw-native-sys/simpler/actions/runs/30082144278/job/89446583483
• https://github.com/hw-native-sys/simpler/actions/runs/30082144278/job/89446583568

The fork token cannot re-run upstream jobs (403 Must have admin rights to Repository). Please repair/clean the a5 runner workspace ownership and re-run these two jobs. No PR code ran in either failure.