@aprilwizard/dsh-multi-cot

English | 中文

Multi-CoT gives dsh an approximation of parallel test-time compute by repeatedly generating and sampling reasoning chains: samples byte-identical calls explore several chains, internal voters pick the best result, and that result drives either a plan-first round or the full three-phase plan → execute → review workflow. The plugin ships only a pure selection core and a stable protocol prompt, hooks into agent/pre-step, llm/stream, and agent/turn-stopping, and requires no dsh source changes.

These recent results come from modifying the codex source directly: a plain vs multi-cot workflow comparison using deepseek-v4-flash in an isolated Terminal-Bench 2.1 run (baseline = plain mode, workflow = multi-sampled three-phase):

Task	baseline	workflow
write-compressor (hard)	✅ 685.6s / 2475B / 1.83M in	✅ 643.5s / 2231B / 1.17M in
cancel-async-tasks (hard)	❌ 219.8s (5/6 tests passed)	✅ 647.3s
polyglot-rust-c (hard)	✅ 522.0s	✅ 1,097.9s
regex-chess (hard)	✅ 1,075.0s / 6.93M in	✅ artifact valid; process killed at 1800s timeout
sqlite-db-truncate (medium)	✅ 129.2s	✅ 598.5s

A single GSM8K problem comparison kept the answer correct (26/26) while latency grew ~29× (4.4s → 126.8s) and input tokens grew 11.1k → ~219k: multi-sampling mostly trades cost for stability.

Note: these are reference results from a recent codex experiment; this dsh plugin has not yet been benchmarked under identical conditions.

Design

Sampling and voting

A normal turn asks the model once and takes whatever it returns. This plugin asks it samples times with identical inputs, then lets samples internal voters score the candidates. Identical inputs keep the calls cheap: providers cache the shared prefix, so parallel samples cost less than the naive multiple. Identical results skip voting.

The workflow

workflow mode runs each task in three phases: information gathering, implementation, and report writing. Each phase has a plan step (offline sampling + voting), an execution step (the ordinary tool loop), and a review step (another offline sampling + voting round). A failed review replans the phase up to reviewMaxRetries times, then forces it forward and marks the result doubtful. The phase machine lives in the plugin, not in model-printed markers.

Where the offline work happens

The plan must be selected before the model request is built. llm/stream returns a stream synchronously, and a plan step cannot claim an empty batch (the loop would close the turn). So:

• the first plan is selected in agent/pre-step;
• later plans are selected in agent/turn-stopping;
• the winner is stored in a process-local table, and llm/stream only short-circuits the armed request with the stored plan stream.

Later-phase plans therefore show one user-role "produce a plan for phase N" line before the plan itself.

Caching

The protocol text is a stable system-prompt section (order 50). Sampling requests are byte-identical. Votes share one prefix — every candidate's compressed chain — and append only the voter's own chain plus the scoring instructions. Measured on both opencode-go and the official DeepSeek API:

Experiment	opencode-go	official DeepSeek
Identical request repeated	0 hits	0 hits
Shared-prefix voters 1+	~90%	~90%
Live workflow cache hits	83.1% (40 calls)	62.7% (32 calls)

Repeating an identical request does not hit the cache (reasoning models put reasoning tokens in the output-end cache unit), so the design relies on shared-prefix reuse.

Reasoning chains

Votes use a compressed version of the model's real reasoning. On the chat-completions wire, reasoning_content arrives as thinking events in pi-ai and becomes reasoning blocks in the dsh adapter. One gotcha: opencode-go's built-in provider defaults to the DeepSeek thinking dialect and sends thinking: {type: "disabled"} without an explicit effort. Set compat.thinkingFormat: openai to keep reasoning on.

Logging and failure

Every offline request, usage, decision, and phase change is logged as a session event, and the chosen plan is written to the transcript by the loop as a normal assistant message, so the session log is the single source of truth. Failures degrade instead of blocking: empty selection → ordinary model call, identical results → skip voting, unusable votes → majority/first, unparseable review → PASS, stream error → normal loop.

Configuration

Key	Default	Meaning
mode	off	off disables; first-plan plans once before execution; workflow runs plan → execute → review for each of the three phases.
samples	3	Number of parallel samples (and internal voters) per offline selection; 2–16 in enabled modes.
votePoints	100	Total integer points one internal vote distributes across samples.
compressedChainMaxTokens	300	Absolute token ceiling for a compressed reasoning chain.
compressedChainRatio	0.2	Relative chain ceiling as a fraction of the full chain.
nearTopDistance	0.05	Fraction of the top score below which candidates stay in the weighted draw.
reviewMaxRetries	2	Failed reviews allowed before a phase is forced to advance.

All values are validated at plugin load; invalid ranges fail loud instead of silently defaulting.

Events

The package declares four log-only SessionEventMap members:

Event	Purpose
multi-cot/phase	Whole-value per-turn phase state (last one wins); resume/fork recover it by folding.
multi-cot/request	Exact system prompt and messages of one offline sample/vote/review request, for reconstructability.
multi-cot/usage	Per-call provider usage of one offline request.
multi-cot/decision	Selected index, normalized scores, and review verdict of one offline selection.

Model Experience

Protocol section

What the model sees

While an enabled mode is composed, every request carries the stable multi-cot:protocol section at prompt order 50.

Workflow mode

The agent completes a task in three phases: information gathering, implementation, and report writing. Each phase runs a plan step, an execution step, and a review step. During execution you may gather missing information directly, but you must not change the plan; when the plan must change, return through review and re-plan. A phase review may fail at most twice before the phase advances anyway; when that happens, mark anything uncertain as doubtful in the final report.

Plan-first mode

Before executing a task, produce one concrete plan, then follow it during execution.

Token effect

Fixed per-request cost while the plugin is composed and the mode is enabled; off contributes nothing.

KV Cache effect

Prefix-stable while the section text and order are unchanged; enabling, disabling, or changing the mode invalidates reuse from order 50 onward.

Plan, execution, and review steps

What the model sees

Plan steps appear as an assistant plan selected offline; later-phase plan steps and executions carry stable user-role request/instruction lines.

Plan request (phases 2+) and execute instruction

Produce a plan for phase 2 (implementation). Do not execute it yet; it will be selected and reviewed before execution.

Execute the plan above for phase 2 (implementation). Gather missing information directly during execution, but do not change the plan unless a review requires it.

Token effect

One extra user-role line before each later-phase plan and one execute instruction per phase; offline sampling and voting tokens appear only in multi-cot/usage events, not in the loop's assistant usage.

KV Cache effect

The offline samples share byte-identical prefixes within a round; the execute instruction appends after the reusable plan prefix.

Known Limitations and Deferred Work

• Offline requests do not participate in context compaction — long sessions could overflow when sampling inputs grow; a compression boundary for offline inputs is deferred.
• Workflow state is not crash-resumed mid-phase — multi-cot/phase folds durable state, but the in-flight plan/review step itself has no checkpoint; an interrupted turn restarts from the next fresh turn.
• No full-chain injection ceiling in votes — the compressed chains are bounded, but the voter's own full chain is injected as captured; an upper bound on that injection is deferred.
• Plan requests for later phases cost one extra user-role line — the plan-request message stays in the transcript before the offline-selected plan; a wake-without-history step is deferred.

Installation

Install the package, then compose it in a cordis.yml:

npm install @aprilwizard/dsh-multi-cot

- id: multi-cot
  name: '@aprilwizard/dsh-multi-cot'
  config:
    mode: workflow   # off | first-plan | workflow
    samples: 3

Peer requirements are @deepseek-ai/cordis, @deepseek-ai/dsh-agent, @deepseek-ai/dsh-invariants, @deepseek-ai/dsh-llm, @deepseek-ai/dsh-session, and @deepseek-ai/dsh-system-prompt.

Install caveat: the published dsh rc.1 packages peer-depend on the unpublished type-only @deepseek-ai/dsh-type-meta, so npm's automatic peer install fails. Until upstream publishes it, install with pnpm (autoInstallPeers: false in pnpm-workspace.yaml) or wait for newer dsh packages.

Development

The plugin is verified against the @deepseek-ai/dsh-* packages published on npm (0.0.1-rc.1 + @deepseek-ai/cordis 4.0.1) — no dsh checkout or link is needed:

pnpm install
pnpm build         # emit lib/types for publishing
pnpm test          # unit + integration + baseline/workflow comparison
pnpm typecheck
pnpm cache-check   # real cache experiments on the opencode-go endpoint
pnpm live-run      # real end-to-end workflow run on the opencode-go endpoint

One install caveat: the published rc.1 dsh packages peer-depend on the type-only @deepseek-ai/dsh-type-meta package, which was never published. pnpm-workspace.yaml therefore sets autoInstallPeers: false and every runtime peer is listed explicitly in devDependencies. Bump the devDependency versions when newer dsh packages publish.

cache-check and live-run read OPENCODE_GO_API_KEY (fallback OPENCODE_API_KEY); neither prints the credential.

Provider note (reasoning capture)

DeepSeek-family models return reasoning_content on the chat-completions wire. pi-ai's built-in opencode-go and deepseek catalog providers default to the DeepSeek thinking dialect, which sends thinking: {type: "disabled"} when no reasoning effort is selected, so reasoning never reaches the harness. Set compat.thinkingFormat: openai to keep the provider's default thinking enabled:

- id: llm
  name: '@deepseek-ai/dsh-llm-pi-ai'
  config:
    providers:
      opencode-go:
        apiKeyEnv: OPENCODE_GO_API_KEY
        api: openai-completions
        baseURL: https://opencode.ai/zen/go/v1
        compat:
          thinkingFormat: openai
        models:
          - id: deepseek-v4-flash
            contextWindow: 1000000
          - id: deepseek-v4-pro
            contextWindow: 1000000
      deepseek:
        apiKeyEnv: DEEPSEEK_API_KEY
        api: openai-completions
        baseURL: https://api.deepseek.com
        compat:
          thinkingFormat: openai
        models:
          - id: deepseek-v4-flash
            contextWindow: 1000000
          - id: deepseek-v4-pro
            contextWindow: 1000000

Both routes are verified with this configuration (official DeepSeek also serves deepseek-chat, which returns no reasoning). No dsh source change is required: with thinkingFormat: openai, pi-ai emits thinking events and the stock dsh adapter maps them to reasoning blocks. The stock adapter's usage mapping does not forward a reasoning-token count, so multi-cot/usage events omit reasoningTokens; chain capture and compression are unaffected.