[Bug] Prime Agent harness: SIGTERM mid-turn freezes the session permanently

[precociousapprentice]

Affected area

Not sure

What happened?

TL;DR: When the Prime Agent worker process is killed (SIGTERM) while a tool call is in-flight, the in-flight call never gets a result. When the session re-attaches, it sits in needs_input on a transcript whose last entry is an unanswered tool call and never resumes the turn — the agent is frozen forever (visible as the "thinking" spinner that never resolves) even though the model is alive. This is a harness-side bug, not an engine bug.

Which component — read this first

1. The inference engine (the local MLX server on 127.0.0.1:8000, the OpenAI-compatible mtplx serve process). This one is healthy. In both incidents it completed every request it received (verified against ~/.mtplx/logs/request-log-8000.jsonl). It is not the bug.
2. The Prime Agent harness (the prime-agent supervisor + per-session worker processes that relay model calls and execute tool calls). This one is the bug. Its worker received SIGTERM mid-turn and never synthesized a terminal tool result, so the re-attached session dangles forever.

The fix belongs in the Prime Agent harness (the worker/shutdown layer), not in the MLX engine. If "MTPLX devs" = the engine team, forward this to the Prime Agent / harness maintainers instead.

The bug (one sentence)

On worker shutdown, the harness must (a) synthesize a terminal tool result for any in-flight tool call, and (b) reap/kill spawned process trees. It does neither — so a SIGTERM mid-call leaves a dangling tool call that a re-attached session treats as "waiting for input" and never resumes.

Steps to reproduce

Reproduction (two confirmed incidents)

Both: model alive, engine healthy, tool call in-flight, worker gets SIGTERM,
session frozen needs_input until the terminal is killed.

Incident A — 2026-08-19 00:06:33Z (full postmortem)

• Worker log: received SIGTERM; shutting down (exit 143) at 00:06:33Z.
• Session frozen needs_input for 2h19m until a terminal restart.
• Full timeline: SESSION-HANDOFF.md → "Harness-incident postmortem".

Incident B — 2026-08-19 16:35:00Z (this investigation)

• Timeline (UTC): 13:00Z clip → cascade (served locally). The session
  (transcript 01a01a20-…) ran ~50 local-only turns (no MTPLX model traffic
  13:00→15:09Z — the engine was idle, not stalled). 15:18:30Z a second
  clip arrived; the cascade ran locally. 15:25:50Z the session issued an
  ipython tool call; its result never arrived. 16:35:00.077Z the
  supervisor received SIGTERM and took the worker(s) down (exit 143);
  the hanging call logged "Request was aborted". 16:37:07Z the supervisor
  re-attached the session → it hung in needs_input forever. 16:52Z the
  user killed the terminal.
• On-disk evidence:
  • ~/.prime/agent/logs/worker-…-7e5fe3fa9241.sock.de451d9d.log:
    received SIGTERM; shutting down (exit 143); closing 1 active session(s)
    at 16:35:00.077Z.
  • ~/.prime/agent/logs/daemon.sock.4260edfa.log: supervisor relaying the
    same SIGTERM to both workers at 16:35:00.078–79Z.
  • Transcript 01a01a20-…jsonl: last assistant entry 15:25:50Z is a
    toolCall (ipython); the next entry 16:35:00Z is a toolResult whose
    text is literally "Request was aborted". No entries after.
  • ~/.mtplx/logs/request-log-8000.jsonl: zero model requests
    13:00→15:09Z, then local-only activity — the engine was idle, not dead.
• Suspect for the SIGTERM sender (unconfirmed): at the exact instant
  (16:35:00Z) a long-running launchd daemon logged a stop/start, and at
  00:06:33Z (incident A) the same class of daemon stopped/restarted
  concurrently. The supervisor is a child of a plain /bin/zsh -il;
  log show records no launchd kill, so the sender is not yet identified.
  (The daemon has since been parked to remove this variable —
  .scratch/llm-wiki-autopipeline/daemon/PARKED.md.)

Expected behavior

The three fixes needed

1. Synthesize a terminal tool result on worker shutdown. When a worker exits mid-call, write a terminal toolResult for the dangling call (e.g. "worker shut down mid-call; turn aborted") so a re-attached session sees a complete tool exchange and resumes (the model can state the abort and continue).
2. Kill spawned process trees on worker shutdown. A nohup / start_new_session child the agent launched outlives the turn and dies orphaned when the worker's session is torn down; the worker should reap the child's tree so pidfiles/logs are left consistent, not half-written.
3. Log the SIGTERM sender. The worker/supervisor log records received SIGTERM but not the sender. Add a sender=<pid> line so these ncidents are diagnosable in minutes, not hours.

What is ruled out

• A model loop (no repeated identical prompts in the MTPLX log).
• A lost model connection (all MTPLX requests in-window completed; the engine was idle, not dead).
• The user's own actions (the 16:52Z kill came 17 min after the freeze).

Prime Agent version

Prime Agent 0.7.2

Environment

MacOS Sequoia 15.7.7, Qwen 3.8 27b, MTPLX haness, terminal in VS Code

Additional context

I am not a real developer. I'm a physician using AI to help me produce a communication app for my patients. Please understand I am doing the best I can. Thank you.