DSH | dsh-llm-gate | One request at a time for local llama.cpp, no more subagent timeouts

[d3vmeh]

Project URL:
https://github.com/d3vmeh/dsh-llm-gate
npm: https://www.npmjs.com/package/dsh-llm-gate

Introduction:

If you run DSH against a local llama-server with --parallel 1, the server can only work on one request at a time. When two requests overlap (the main agent and a subagent, or the agent and compaction), llama.cpp quietly puts the second one on hold and sends nothing back. DSH cannot tell "waiting" from "dead", so after 5 minutes it gives up with terminated or pi-ai stream idle timeout, then retries the whole prompt up to five times.

dsh-llm-gate fixes this by making DSH wait its turn. Extra requests are held inside DSH, in order, and only sent to llama.cpp when the server is free. The main agent and subagents then take turns instead of one of them failing.

How it integrates with DSH:

It is a small cordis plugin that hooks the point where DSH sends any model request (llm/stream), so it covers everything: agents, subagents, compaction, and session titles. Each request takes a slot before it is sent and gives it back when its stream ends, whether that is normally, by error, or by cancellation. If the queue is full or a request waits too long, it fails with a clear QUEUE_FULL or QUEUE_TIMEOUT error instead of being retried. Installation is one command:

dsh plugin --profile web add dsh-llm-gate

then in ~/.dsh/profiles/web/cordis.patch.yml:

- id: llm-gate
  config:
    providers:
      llamacpp:
        maxConcurrent: 1          # match llama.cpp --parallel
        maxQueued: 16             # optional
        queueTimeoutMs: 3600000   # optional

What it looks like in use:

Main agent and a background subagent sharing one llama.cpp slot. The lines only appear when a request had to wait:

llm-gate: llamacpp session=a61e6e40 queued (depth 1)
llm-gate: llamacpp session=a61e6e40 dispatched after 5730ms
llm-gate: llamacpp session=bd122c6d queued (depth 1)
llm-gate: llamacpp session=bd122c6d dispatched after 30930ms
llm-gate: llamacpp session=a61e6e40 queued (depth 1)
llm-gate: llamacpp session=a61e6e40 dispatched after 30886ms

Two related findings, independent of the plugin:

• For long prompts (200K tokens can be 30 to 60 minutes of prefill on a 27B model), set streamIdleTimeoutMs on the provider route. llama.cpp's SSE pings keep the HTTP connection alive during prefill, so that setting fixes the single-request case. Consider removing TIMEOUT from retryPolicy.retryableCodes so a genuine timeout costs 5 minutes, not 30.
• The gate serializes; it does not make one slot faster. For real parallelism use --parallel 2 --kv-unified and maxConcurrent: 2.

MIT License. Feedback welcome!

> Unofficial project, independently developed and maintained by community members.