Repository navigation
v1.15.12
Two-commit hardening of the wedged-job watchdog, driven by production incidents on the LTX deployment (6495a152e2):
Observed failure: a job wedges (ComfyUI threads stuck in uninterruptible NFS page-fault waits — GPU 0%, VRAM held), /ready 503s, the watchdog logs "exiting to trigger redeploy" — and then the process stays alive (one replica: 17 hours). The container never recycles; the fleet accumulates zombie workers. Separately observed: the watchdog firing hours late because cgroup memory-throttle starves the event loop.
3024bce (1.15.11, previously unpushed): SIGKILL the ComfyUI process group before exiting — releasing its memory is what un-throttles the cgroup enough for the proxy to die.
this commit (1.15.12): replace process.exit(1) with self-SIGKILL. exit still needs the throttled JS runtime to be scheduled, and killing ComfyUI doesn't always release memory (pages pinned in unkillable hard-mount NFS waits). Kernel signal delivery needs no JS. The logger is flushed first so the fatal line survives.
Note for ops: an in-process watchdog can still fire late under heavy throttle — an AutoDL-side health check on /ready with auto-restart remains the backstop for fully-frozen replicas (one was observed where even fork hung).
🤖 Generated with Claude Code