fix: reap the MPExecutor process group to free GPU on task failure

[kaiitunnz]

Purpose

When an executor hosted in MPExecutor failed with an unexpected (non-ExecutionError) exception after loading its engine — e.g. a vLLM inference task that loads onto the GPU and then raises during prompt rendering — the worker hard-killed only the direct subprocess. vLLM's engine-core child is a grandchild of that subprocess, so it was orphaned and kept holding the GPU. The next task on that worker then OOMed at model load. This is a regression from the unexpected-failure teardown added in #82, which terminated only the direct child.

Changes

• src/worker/executors/mp_executor.py — on an unexpected failure, shut the subprocess down gracefully while it is still responsive, then reap its whole process group on teardown so an orphaned engine-core child is killed rather than leaked.
• tests/worker/test_mp_executor_lifecycle.py — cover graceful inner cleanup on an unexpected failure, and process-group reaping of an orphaned grandchild.

Design

The worker subprocess leads its own process group (os.setsid()), so teardown signals the group — reaching the engine-core grandchild vLLM spawns, not just the direct child. When the subprocess is still responsive, a graceful shutdown runs the inner executor's cleanup first so the GPU is released cleanly; the group kill is the backstop and also covers the OOM-killed-subprocess path, where graceful shutdown isn't possible.

Test Plan

uv run pytest tests/ --ignore=tests/worker/test_mp_executor_cleanup_gpu.py
uv run pre-commit run --all-files

End-to-end on a local root + single GPU worker on freshly built images carrying this branch: submit a vLLM inference task that loads its engine and then fails during graph-template rendering, then submit a follow-up vLLM task on the same worker — comparing the worker's GPU state and the follow-up's outcome with and without the fix. GPU memory is attributed per worker container (the worker's own processes) so the reading is robust to other tenants sharing the device.

Test Result

$ uv run pytest tests/ --ignore=tests/worker/test_mp_executor_cleanup_gpu.py   # 1139 passed
$ uv run pre-commit run --all-files   # gitleaks / isort / black / ruff / codespell / mypy / sync-requirements passed

End-to-end: without the fix, the failed task left an orphaned vLLM engine-core process holding ~5.7 GiB on the worker, and the follow-up task failed with vLLM's No available memory for the cache blocks. With the fix, the worker's GPU footprint returned to 0 after the failure and the follow-up task ran to DONE.

---

Pre-submission Checklist

• I have read the contribution guidelines.
• I have run pre-commit run on the changed files and fixed any issues.
• I have added or updated tests covering my changes (if applicable).
• I have verified that the worker test suite passes locally (worker-only change).
• If I changed shared schemas or proto definitions, I have checked downstream compatibility across Server and Worker. (No schema/proto changes.)
• If I changed the SDK or CLI, I have verified the affected packages work. (No SDK/CLI changes.)
• If this is a breaking change, I have prefixed the PR title with [BREAKING] and described migration steps above.
• I have updated documentation or config examples if user-facing behavior changed. (No user-facing config/behavior change.)

[timzsu]

LGTM. Thanks for the prompt fix.