Skip to content

bug: a stalled agent session hangs the chat task forever, and the whole transcript is lost if you kill it #213

Description

@ThalynLabs

Summary

If an agent session stops without emitting a terminal event, the chat task waits forever. There is no timeout anywhere in the path. The message stays done=0, the UI shows a task in progress indefinitely, and — because streamed text is only persisted when the run completes — the entire transcript is unrecoverable if the user kills the process to escape.

Separately, the error that caused the stall is silently discarded, so nothing anywhere explains what happened.

Observed on an opencode profile, but the timeout gap is structural rather than adapter-specific.

Environment

  • cptr 0.9.20
  • opencode 1.18.5
  • macOS, Python 3.11, host install via uvx

What happened

A long agent run executed normally for about a minute (multiple read and bash tool calls), then stopped dead. It sat for 49 minutes before I investigated. From the outside it was indistinguishable from a model still working.

State at that point:

  • agent server process alive, 0.1% CPU, no child processes
  • no outbound connection from the model provider — the upstream call had long since ended
  • chat message: done=0, content empty, output null
  • last streamed-text flush in the server log: 49 minutes earlier

Querying the agent server directly showed the cause — the final assistant message ended on a tool part still marked running:

{"status": "running",
 "input": {"command": "grep -n \"...\" \"$SCRATCH/...\" | head",
           "workdir": "/Users/u/GitHub/YitHub/project"}}

The model had emitted a corrupted workdir (a mangled path segment) that does not exist. The spawn failed, and the tool part never left running. No terminal event was ever emitted.

Why it hangs forever

In cptr/utils/agents/opencode.py:

  • _collect_opencode_events only puts the sentinel that ends the run when it sees session.status with status.type == "idle". A session that stalls without reaching idle never produces it.
  • run_opencode_agent then blocks on await event_queue.get() with no timeout and no watchdog.
  • The HTTP client is constructed as httpx.AsyncClient(base_url=server_url, timeout=None)explicitly unbounded.
  • The only asyncio.wait_for calls in the module guard server startup and process teardown. Neither covers the event stream.
  • cptr/utils/chat_task.py imposes no per-task timeout either.

So there is no layer that can end a stalled run.

Why nothing is diagnosable

async def _drain_stderr(proc):
    while await proc.stderr.readline():
        pass

The agent process's stderr is read and thrown away. Whatever the tool layer reported about the failed spawn was discarded, which is why neither the UI nor the server log nor the database contained any hint.

The severity is in the data loss, not the wait

Streamed text is accumulated in memory and written on AgentDone. Resume state — chat.meta.agent_sessions[<profile>].session_id — is also only written then. So while the task is hung:

  • the transcript exists nowhere on disk
  • no resume handle exists

The instinctive fix (kill the server, restart) therefore destroys the whole run. In my case that was 29 messages and 28 completed tool calls.

Recovery that does work

Posting an abort to the agent session — rather than killing anything — makes the agent emit its terminal events. cptr then completes normally: it flushed 4668 characters / 72 output items into the message and wrote the resume handle. The session survived on disk, a fresh server found all 29 messages, and a resumed prompt had the model correctly recall its exact position mid-task.

Worth documenting regardless of the fix, since the intuitive recovery is the destructive one.

Suggested fixes

  1. A watchdog on the event stream. If no event of any kind arrives for N seconds, emit AgentError instead of waiting forever. A generous default (several minutes) preserves long tool calls while bounding the failure.
  2. Give the HTTP client a real read timeout rather than timeout=None, at least on the event stream.
  3. Stop discarding stderr. Capture it and include the tail in the AgentError. This one change would have made the incident self-explanatory.
  4. Persist streamed text incrementally, or write resume state as soon as the session id is known, so an interrupted run is recoverable rather than lost.
  5. Optionally, treat a tool part sitting in running with no further events as a terminal condition — that is the concrete shape this took.

Items 1 and 3 are individually sufficient to turn a silent 49-minute hang into an immediate, readable failure.

Related

#212 — a different defect in the same adapter (prompt/abort routes built from OpenAPI operationIds). Independent of this one: that fails loudly and immediately, this fails invisibly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions