fix: recover stuck workers when an MPExecutor subprocess dies

[kaiitunnz]

Purpose

A worker would get stuck BUSY indefinitely when an executor running inside MPExecutor (e.g. vLLM) died hard — an out-of-memory kill, a CUDA abort(), or a native segfault. MPExecutor.run() waited for the subprocess's result with an unbounded res_q.get(), but a child that dies that way never posts a result, so the parent thread blocked forever while holding the executor lock. run() never returned, the runner's finally: set_idle() never ran, and the worker stayed BUSY (with stale=false, so no health/staleness path reclaimed it) — silently unable to accept further work.

Changes

• src/worker/executors/mp_executor.py — wait for the subprocess result with liveness-checked polling instead of an unbounded blocking get, so a dead child surfaces as a retryable failure rather than a hang; restart the subprocess after an unexpected failure to drop a possibly-corrupted engine; share the teardown logic across the run, await, and cleanup paths.
• tests/worker/test_mp_executor_lifecycle.py — cover the dead-child, unexpected-failure, and controlled-error paths.

Design

run() now classifies how a task ended and reacts accordingly: a child that died silently is reported as a retryable error (a hard crash is usually transient, e.g. OOM under contention); a controlled ExecutionError keeps the subprocess warm to avoid a needless engine reload; an unexpected exception restarts the subprocess, since a corrupted engine's GPU memory is only reclaimed once the process exits.

Test Plan

uv run pytest tests/ --ignore=tests/worker/test_mp_executor_cleanup_gpu.py
uv run pre-commit run --all-files

End-to-end on a local root + single GPU worker on freshly built images carrying this branch, using a fault-injection repro: submit a vLLM inference task, then kill -9 the MPExecutor child subprocess mid-run to faithfully simulate an OOM kill / CUDA abort, and observe the worker and task state before and after the fix.

Test Result

$ uv run pytest tests/ --ignore=tests/worker/test_mp_executor_cleanup_gpu.py   # All passed
$ uv run pre-commit run --all-files   # isort / black / ruff / mypy / codespell / gitleaks passed

End-to-end: before the fix, killing the child left worker=BUSY and task=DISPATCHED for the full 120s observation window. After the fix, the dead child is reported as mp(vllm) subprocess exited (code -9), the task fails retryably, is re-dispatched, and runs to DONE, and the worker returns to IDLE.

---

Pre-submission Checklist

• I have read the contribution guidelines.
• I have run pre-commit run on the changed files and fixed any issues.
• I have added or updated tests covering my changes (if applicable).
• I have verified that the worker test suite passes locally (worker-only change).
• If I changed shared schemas or proto definitions, I have checked downstream compatibility across Server and Worker. (No schema/proto changes.)
• If I changed the SDK or CLI, I have verified the affected packages work. (No SDK/CLI changes.)
• If this is a breaking change, I have prefixed the PR title with [BREAKING] and described migration steps above.
• I have updated documentation or config examples if user-facing behavior changed. (No user-facing config/behavior change.)

[timzsu]

LGTM.