Sessions permanently stuck after server restart or stream interruption — no startup recovery for orphaned messages/tool parts

[dzianisv]

Description

When the OpenCode server restarts (or the process crashes) while a session is actively executing tool calls, the session gets permanently stuck in a "Thinking" state. The root cause is that there is no startup recovery that cleans up orphaned assistant messages and tool parts.

What happens

1. A session is actively executing tool calls (e.g., bash commands)
2. The server restarts or crashes
3. The in-memory session state (SessionStatus) is lost — the session is no longer "busy"
4. But the database state is stale: the last assistant message has time.completed = undefined (never completed) and tool parts remain in status: "running" forever
5. The UI sees the incomplete assistant message and shows a permanent "Thinking" spinner
6. The session cannot recover — sending a new message creates a new loop iteration, but the old orphaned message still exists

Root cause analysis

The existing cleanup in processor.ts:402-417 correctly handles the normal case — when the stream ends (normally, via error, or abort), it force-sets any non-terminal tool parts to status: "error". However, this cleanup only runs if the process survives long enough to reach it.

There is zero recovery at startup:

• Session.initialize() does not scan for orphaned messages
• SessionStatus (in-memory map) is empty after restart — no stale detection
• No background watchdog checks for sessions stuck in busy state

The only defense is in toModelMessages() (message-v2.ts:740-746), which converts pending/running tool parts into "[Tool execution was interrupted]" when building the next LLM prompt. This helps contextual recovery if the user sends a new message, but the UI still shows the session as stuck because the orphaned assistant message has no time.completed.

Observed in production

• Session ses_2f4299f5cffeVZfxCt3ViZ7eVJ stuck for 3+ hours with a git log tool part permanently in "running" status
• Session ses_2e9127723ffeKJ1JpjLNS35B4z similar pattern (though this one was actually still running a long k8s test — but demonstrates the same vulnerability)

Relation to existing issues

This is the backend root cause behind several reported symptoms:

• Web UI shows permanent Thinking spinner after stream interruption or server restart #17680 — Web UI permanent Thinking spinner after stream interruption
• QUEUED tag stuck on all user messages when an assistant message has null time.completed #16856 — TUI stuck QUEUED badges from orphan assistant messages
• Intermittent hang: session stays running forever until manual interrupt #14769 — Intermittent hang: session stays running forever
• Tasks/Subagents with Codex / OpenAI are frequently getting stuck with no timeout/retry, which then hangs the session forever #11865 — Subagents stuck with no timeout/retry
• Explore subagent hangs indefinitely with Anthropic Claude Opus 4.6 -- no timeout or recovery #13841 — Explore subagent hangs indefinitely with no recovery

Open PRs #16907 and #17593 address frontend symptoms (making the UI more defensive about stale state), but neither fixes the backend root cause — orphaned messages and tool parts in the database.

Proposed fix

Startup recovery in Session or app bootstrap:

1. On server start, query all messages where time.completed IS NULL and the message role = "assistant"
2. For each orphaned message:
   • Set time.completed = Date.now()
   • Set all tool parts with status = "running" or status = "pending" to status = "error" with error = "Tool execution was interrupted (server restart)"
   • Emit Bus events so connected frontends update

This is a small, safe change — the cleanup logic already exists in processor.ts:402-417, it just needs to be callable from a recovery path at startup.

Steps to reproduce

1. Start opencode serve
2. Start a session that uses tool calls (e.g., ask it to run tests)
3. Kill the server process while tools are executing (kill -9)
4. Restart the server
5. Open the session in the UI — it shows permanent "Thinking" spinner
6. Session status API returns {} (idle) but the UI is stuck

Environment

• opencode serve (long-running, multiple sessions)
• macOS / Linux
• Any provider (observed with gpt-5.3-codex via github-copilot)

OpenCode version

Latest dev branch (commit 814a515a8)

[dzianisv]

We hit the same issue downstream in dzianisv/opencode.\n\nOur merged frontend mitigation is here: https://github.com/dzianisv/opencode/pull/57\n\nThat fix avoids sessions appearing permanently busy by only checking the last message in the UI instead of scanning older orphaned assistant messages.\n\nWhile tracing the root cause, we found the backend can still leave assistant messages without in crash/interruption paths (for example process shutdown or exceptions after stream setup), so the frontend fix is only a mitigation. A backend recovery/finalization fix is still needed upstream.