fix(v1): detect env-server worker death instead of hanging

[eligotts]

Problem

An env server that dies is invisible to its client — the failure shows up as a wait, not an error. Reported from prime-rl training: "environment processes dying on startup but the orchestrator continues to wait for them."

Three separate gaps, each of which converts a crash into an unbounded wait:

1. Readiness meant "socket bound", not "env loaded". serve_env put the pool's address on address_queue before pool.run() spawned any worker:

pool = EnvServerPool(...)          # binds the frontend
address_queue.put(pool.address)    # ← reported "ready" here
asyncio.run(pool.run())            # ← only now spawns worker 1, which loads the env

So a spawner was told "ready" about a pool whose env was still loading, or already dead. An env that fails to import (bad dependency, missing credentials) surfaced much later as a rollout that never returns. This is the default path for prime-rl, whose pool config defaults to elastic.

2. A dead worker was never reaped. Its active count never drained, so least-busy dispatch (min(self.workers, key=...active)) elected the corpse for every subsequent request — a black hole, not a single dropped request. Its pending entries were never resolved, and EnvClient awaits rollout replies untimed by design, so the client waited forever. The module docstring acknowledged this as a TODO on the assumption that "worker death is rare".

3. run_eval_server read the address queue blind. address_queue.get(timeout=600) with no liveness check: a server that died on startup was reported as a 600s timeout rather than a crash, with the real traceback nowhere in the error.

Changes

• Readiness is a worker handshake. EnvServerPool takes the address_queue and reports its address only after a worker answers health — a worker binds its socket before loading the env, so its first reply is what proves the env loaded. Untimed on purpose: a slow load (a legacy dataset pull) is the spawner's deadline to enforce; the broker only insists the process is still alive.
• Worker death is reaped. The serve loop polls worker exit codes (rate-limited to 250ms — reading exitcode is a waitpid per worker and the loop turns once per request), retires the dead worker from dispatch, and answers its in-flight requests with BaseResponse(success=False, ...) so the client raises instead of hanging. Requests arriving with no live workers are failed immediately rather than queued into the void.
• Dead workers are replaced, bounded by MAX_WORKER_RESTARTS = 3: enough to ride out a one-off crash (an OOM, a segfaulting sandbox), few enough that an env dying on every load fails loudly instead of respawning forever.
• wait_for_address(process, address_queue, ...) is the shared queue reader: it polls the process's exit code alongside the queue and quotes the tail of the server's log file in the error. Used here by run_eval_server; prime-rl's orchestrator uses it for the servers it spawns (docs(algo): describe shipped DPPO masking in place of stale importance-ratio clamp prime-rl#3135).

run()'s body is split into _on_request / _on_reply / _reap_dead_workers so the loop stays readable; in_flight and pending moved onto the instance to avoid threading them through. The wire protocol is unchanged, so EnvClient needs no changes.

Verification

Existing suites pass (tests/test_env_server.py, tests/v1/test_configs.py, tests/test_imports.py). Behaviour was checked with a throwaway script (not committed, per AGENTS.md) covering four cases against a real broker and real worker processes:

case	before	after
worker dies loading the env (run())	reports ready, hangs later	raises in 1.0s
server dies on startup (wait_for_address)	600s timeout, cause unreported	raises in 2.0s with exit code + log tail
healthy pool (regression)	ready	ready in 1.5s, serves
worker killed with a request in flight	hangs forever	request fails in 0.3s; pool recovers in 0.7s via a replacement

The last case suspends the worker so the request is genuinely in flight, then SIGKILLs it — the black-hole scenario from gap 2.

Review follow-ups (25bd48d)

Two bugs in the death handling itself, both found by review and both real:

• _replace_worker called _spawn_worker unguarded. A worker death caused by memory pressure is exactly when spawning its replacement can fail, and that exception escaped the serve loop — killing the broker and stranding every other client's in-flight requests. Mid-serve spawns now go through _try_spawn_worker, which logs and leaves the pool one worker short. _maybe_scale_up had the same latent bug (pre-existing) and takes the same path. Startup spawns still propagate: a pool with no workers has nothing to serve.
• Reaping closed a dead worker's socket after relaying only one reply, so a worker that answered several requests and then exited had the rest discarded and reported as worker died — silently relabelling successful rollouts as failures. _reap_dead_workers now drains the socket before retiring it. The serve loop still takes one reply per pass so a chatty worker can't starve the frontend.

Two things to look at in review:

• run_eval_server no longer calls client.wait_for_server_startup() after reading the address. It is now redundant — the address is the readiness signal — and it never meant anything for a pooled server anyway, since the broker answers health inline whatever its workers are doing. But the --server eval path needs live models, so I could not run it locally; only the pool-level behaviour above was exercised.
• MAX_WORKER_RESTARTS is a per-pool budget, not per-worker, so a large pool losing one worker each to OOM degrades after three. That seemed the right default over per-worker budgets (which never stop respawning a systematically broken env), but it's a judgement call.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes core broker dispatch, readiness, and failure paths for training/eval orchestration; wire protocol unchanged but rollout failures may surface as explicit errors where clients previously hung.

Overview
Fixes env-server crashes being reported as indefinite waits by tightening readiness, startup waiting, and worker lifecycle in the v1 serve pool.

Readiness no longer means the broker socket is bound. EnvServerPool puts its address on address_queue only after a worker responds to a startup health probe (env actually loaded). serve_env passes the queue into the pool instead of reporting immediately after bind.

Startup uses new wait_for_address, which polls the spawned process exit code while waiting on the queue and includes a log tail on failure. run_eval_server adopts this and drops the redundant EnvClient.wait_for_server_startup() call.

Runtime adds periodic reaping of dead workers: drain queued replies, fail orphaned in-flight requests with BaseResponse(success=False), retire the worker, and spawn replacements up to MAX_WORKER_RESTARTS. Mid-serve spawns go through _try_spawn_worker so spawn failures do not kill the broker. Requests with no live workers fail immediately. The broker loop is refactored into _on_request / _on_reply / _reap_dead_workers with instance-level pending and in_flight.

^{Reviewed by Cursor Bugbot for commit f0c87f2. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Detect env-server worker death during startup instead of hanging indefinitely

• Replaces the bare address_queue.get + wait_for_server_startup calls in runner.py with wait_for_address, which polls the child process exit code and raises immediately if it dies.
• Adds wait_for_address and ENV_SERVER_SPAWN_TIMEOUT to serve/pool.py; both are now publicly exported from verifiers.v1.serve.
• The pool broker now gates its 'ready' signal on a worker answering a health probe, detects and reaps dead workers, fails their in-flight requests with a structured error, and attempts bounded automatic replacement up to MAX_WORKER_RESTARTS.
• Health requests are handled inline by the broker; non-health requests return an error when no live workers exist.
• Behavioral Change: broker startup no longer reports ready immediately on socket bind — it waits for a worker to respond to a health probe first.

^{Macroscope summarized f0c87f2.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces substantial new runtime behavior for worker death detection and replacement (~250 lines of new lifecycle management logic). Two unresolved review comments identify potential bugs: a high-severity issue where scale-up bypasses the restart budget, and a medium-severity readiness hang in the single-worker path.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f0c87f2. Configure here.}

[cursor]

Scale-up bypasses restart budget

High Severity

_maybe_scale_up spawns workers without consulting MAX_WORKER_RESTARTS or _restarts. After the replacement budget is spent, sustained load can keep adding workers through scale-up, recreating the unbounded respawn loop that budget was meant to stop — especially with default elastic max_workers=None.

Additional Locations (1)

• verifiers/v1/serve/pool.py#L348-L350

 

^{Reviewed by Cursor Bugbot for commit f0c87f2. Configure here.}

[cursor]

Single-worker readiness hang returns

Medium Severity

Removing wait_for_server_startup leaves the max_workers=1 path without a readiness check. That path still publishes the address from run_server before run()/serving(), and wait_for_address returns as soon as the queue yields, so a death right after publish succeeds the wait and the client hangs on untimed requests.

Additional Locations (1)

• verifiers/v1/serve/pool.py#L551-L553

 

^{Reviewed by Cursor Bugbot for commit f0c87f2. Configure here.}