DSH | dsh-turn-doctor | Find out which timeout actually killed a failed turn

[d3vmeh]

Project URL:
https://github.com/d3vmeh/dsh-turn-doctor

npm:
https://www.npmjs.com/package/dsh-turn-doctor

Introduction

This came out of a problem that's been mentioned a few times in DSH discussions.

Back in #3157, someone put it pretty well:

"there seem to be multiple timeouts involved, and it is hard to tell which one actually killed the turn... Would it be possible to log the actual cancellation source somewhere?"

That's basically what dsh-turn-doctor does.

A request can fail in several different places: DSH's idle watchdog (streamIdleTimeoutMs), the SDK request timer (timeoutMs), Node/undici's HTTP timers, the model server itself, or an individual tool call.

The annoying part is that the error shown in the UI often isn't enough to tell those apart. Request timed out. can come from more than one timer. terminated might mean undici stopped waiting for the response body, or it might mean the model server died.

So you can end up increasing one timeout, retrying, and still having the request die at exactly 5:00. That's also come up in #4518.

dsh-turn-doctor puts its own stopwatch around each model request and tracks things like time to first byte, longest period of silence, and total request time. When something fails, it combines those measurements with the actual error and tries to identify which layer caused it.

For example:

turn-doctor: session=... turn 1 attempt 1
  verdict: Node's HTTP headers timer (undici) gave up waiting for the server's first response after 300.6 s.
  evidence: no reply bytes in 300.6 s
  fix: install dsh-fetch-timeouts to raise Node's HTTP timers; on llama.cpp this usually means the request was deferred behind a busy slot

It also catches a couple of failures I wasn't getting much visibility into before, especially failed compactions and tool timeouts. In my own logs I found six compaction summaries that had quietly hit their token cap.

You can type /why in chat to see the recent diagnoses for that session, including failures from subagents.

How it integrates with DSH

It's a host-side Cordis plugin.

It observes the llm/stream waterfall to collect timing data without modifying the chunk stream, plus agent/request-error and a few session events such as llm/retry, turn/end, compaction/end, and tool/result.

It doesn't retry requests or change any DSH behavior. It's just watching and reporting.

Verdicts currently show up in:

• the DSH terminal
• /why
• turn-doctor/verdict events in the session log

That last one should also make it possible to add a proper row for these in the web UI later.

I don't duplicate retry countdowns since DSH already renders those.

Install:

dsh plugin --profile web add dsh-turn-doctor

It works without any config and assumes DSH's default timeout values.

If you've changed your route timeouts, you can tell the plugin what they are so it can make better timing-based diagnoses:

- id: turn-doctor
  config:
    providers:
      ollama:
        streamIdleTimeoutMs: 1800000
        undiciTimeoutMs: 1800000   # if dsh-fetch-timeouts is installed

A few notes

• If the timing isn't enough to distinguish two possible causes, it says that rather than making up a confident answer.
• I've tested it end to end against stub servers for things like silent stalls and connection refusals, plus a real llama.cpp context overflow.
• There are currently 71 unit tests replaying the failure cases I've collected from my own session logs.
• dsh-error-lens is related, but takes a different approach. It shows recent failures in a web panel grouped by HTTP status. dsh-turn-doctor is more focused on separating the different timeout layers using timing data, and it also covers compactions and tool calls.

This is also the fourth plugin in the little local-model setup I've been building around DSH:

• dsh-llm-gate (DSH | dsh-llm-gate | One request at a time for local llama.cpp, no more subagent timeouts #4995) keeps concurrent requests within the model server's available slots
• dsh-context-budget (DSH | dsh-context-budget | Keep a local model's context at a size your GPU handles well #5078) keeps context usage under control
• dsh-fetch-timeouts (DSH | dsh-fetch-timeouts | Stop slow local models (Ollama, LM Studio) from being cut off at 5 minutes #5124) lets you get past Node's 5-minute HTTP cutoffs
• dsh-turn-doctor tries to tell you what actually went wrong when a turn still fails

MIT licensed. Would appreciate feedback, especially if anyone has failure cases it misclassifies or doesn't recognize yet.

> Unofficial project, independently developed and maintained by community members.