Skip to content

health_check raises ValueError: process object is closed after inference subprocess exits (missing same guard as _send_kill_signal, see #4500/#4694) #6672

Description

@jeser-aj

Bug Description

worker.py's health_check HTTP handler calls self._inference_executor.is_alive() without guarding against the underlying multiprocessing.Process already being closed. Once the shared inference subprocess exits, SupervisedProc._supervise_task() calls self._proc.close() (supervised_proc.py:300). Any health_check request that arrives after that point raises an unhandled ValueError: process object is closed, which propagates out of the handler as a 500 instead of the intended 503.

This is the same race already fixed in _send_kill_signal() (see #4500, #4694) — that fix wrapped the identical self._proc.is_alive() call in try/except ValueError, but health_check() in worker.py was never updated with the same guard.

Stack Trace

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/multiprocessing/process.py", line 101, in _check_closed
    raise ValueError("process object is closed")
ValueError: process object is closed

  File "livekit/agents/ipc/inference_proc_executor.py", line 104, in is_alive
    return self._proc.is_alive()
  File "livekit/agents/worker.py", line 363, in health_check
    if self._inference_executor and not self._inference_executor.is_alive():
  File "aiohttp/web_protocol.py", line 510, in _handle_request
    resp = await request_handler(request)

Environment

  • livekit-agents 1.2.18 (confirmed still present, unfixed, in latest 1.6.7)
  • Python 3.11
  • Linux (k8s)

Steps to Reproduce

  1. Run a worker with a plugin that registers an InferenceRunner (e.g. livekit.plugins.turn_detector.english.EnglishModel), which spawns a shared inference subprocess for the worker.
  2. Cause the inference subprocess to exit (crash, OOM, unresponsive ping/pong timeout, etc.) without the whole worker process restarting.
  3. Once _supervise_task() runs self._proc.close(), every subsequent hit to the worker's / health-check HTTP endpoint raises ValueError: process object is closed instead of returning a 503.
  4. This repeats on every health-check poll indefinitely (e.g. every ~1 min from a k8s probe), since nothing recreates the inference subprocess for that worker instance.

Expected Behavior

health_check() should treat a closed inference process the same as a dead one and return 503 inference process not running, not raise an unhandled exception.

Suggested Fix

Same pattern already used in _send_kill_signal():

async def health_check(_: Any) -> web.Response:
    try:
        inference_alive = not self._inference_executor or self._inference_executor.is_alive()
    except ValueError:
        inference_alive = False

    if not inference_alive:
        return web.Response(status=503, text="inference process not running")

    return web.Response(text="OK")

Impact

  • Every health-check poll after the inference subprocess dies throws an unhandled exception and returns 500 instead of 503, which can confuse liveness/readiness probes and floods error monitoring (we saw ~20k occurrences in Sentry over 2 days from a single worker pod).
  • No indication in the code that a dead shared inference subprocess is ever respawned within a running worker instance, so this is a persistent failure mode until the pod is restarted.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions