Bug Description
worker.py's health_check HTTP handler calls self._inference_executor.is_alive() without guarding against the underlying multiprocessing.Process already being closed. Once the shared inference subprocess exits, SupervisedProc._supervise_task() calls self._proc.close() (supervised_proc.py:300). Any health_check request that arrives after that point raises an unhandled ValueError: process object is closed, which propagates out of the handler as a 500 instead of the intended 503.
This is the same race already fixed in _send_kill_signal() (see #4500, #4694) — that fix wrapped the identical self._proc.is_alive() call in try/except ValueError, but health_check() in worker.py was never updated with the same guard.
Stack Trace
Traceback (most recent call last):
File "/usr/local/lib/python3.11/multiprocessing/process.py", line 101, in _check_closed
raise ValueError("process object is closed")
ValueError: process object is closed
File "livekit/agents/ipc/inference_proc_executor.py", line 104, in is_alive
return self._proc.is_alive()
File "livekit/agents/worker.py", line 363, in health_check
if self._inference_executor and not self._inference_executor.is_alive():
File "aiohttp/web_protocol.py", line 510, in _handle_request
resp = await request_handler(request)
Environment
livekit-agents 1.2.18 (confirmed still present, unfixed, in latest 1.6.7)
- Python 3.11
- Linux (k8s)
Steps to Reproduce
- Run a worker with a plugin that registers an
InferenceRunner (e.g. livekit.plugins.turn_detector.english.EnglishModel), which spawns a shared inference subprocess for the worker.
- Cause the inference subprocess to exit (crash, OOM, unresponsive ping/pong timeout, etc.) without the whole worker process restarting.
- Once
_supervise_task() runs self._proc.close(), every subsequent hit to the worker's / health-check HTTP endpoint raises ValueError: process object is closed instead of returning a 503.
- This repeats on every health-check poll indefinitely (e.g. every ~1 min from a k8s probe), since nothing recreates the inference subprocess for that worker instance.
Expected Behavior
health_check() should treat a closed inference process the same as a dead one and return 503 inference process not running, not raise an unhandled exception.
Suggested Fix
Same pattern already used in _send_kill_signal():
async def health_check(_: Any) -> web.Response:
try:
inference_alive = not self._inference_executor or self._inference_executor.is_alive()
except ValueError:
inference_alive = False
if not inference_alive:
return web.Response(status=503, text="inference process not running")
return web.Response(text="OK")
Impact
- Every health-check poll after the inference subprocess dies throws an unhandled exception and returns 500 instead of 503, which can confuse liveness/readiness probes and floods error monitoring (we saw ~20k occurrences in Sentry over 2 days from a single worker pod).
- No indication in the code that a dead shared inference subprocess is ever respawned within a running worker instance, so this is a persistent failure mode until the pod is restarted.
Bug Description
worker.py'shealth_checkHTTP handler callsself._inference_executor.is_alive()without guarding against the underlyingmultiprocessing.Processalready being closed. Once the shared inference subprocess exits,SupervisedProc._supervise_task()callsself._proc.close()(supervised_proc.py:300). Anyhealth_checkrequest that arrives after that point raises an unhandledValueError: process object is closed, which propagates out of the handler as a 500 instead of the intended 503.This is the same race already fixed in
_send_kill_signal()(see #4500, #4694) — that fix wrapped the identicalself._proc.is_alive()call intry/except ValueError, buthealth_check()inworker.pywas never updated with the same guard.Stack Trace
Environment
livekit-agents1.2.18 (confirmed still present, unfixed, in latest 1.6.7)Steps to Reproduce
InferenceRunner(e.g.livekit.plugins.turn_detector.english.EnglishModel), which spawns a shared inference subprocess for the worker._supervise_task()runsself._proc.close(), every subsequent hit to the worker's/health-check HTTP endpoint raisesValueError: process object is closedinstead of returning a 503.Expected Behavior
health_check()should treat a closed inference process the same as a dead one and return503 inference process not running, not raise an unhandled exception.Suggested Fix
Same pattern already used in
_send_kill_signal():Impact