[Proposal] Provider-neutral prompt token pressure + local overflow -> compact-retry

[tianyiming1]

Status: community plugin available now (dsh-plugin); core protocol sketched for when external PRs are accepted again.
Related: CONTRIBUTING currently declines external PRs — filing this as a Discussion per project guidance.

Problem

Stock dsh-token-meter prices prompts with a fixed ~4 characters/token heuristic. Dense-script (CJK) sessions and large tool schemas are systematically undercounted. Local OpenAI-compat servers (e.g. llama.cpp) then reject the real request with:

exceed_context_size_error
request (N tokens) exceeds the available context size (n_ctx)

Two gaps follow:

1. Proactive compaction trusts the undercount → pressure never fires in time.
2. Overflow recovery already exists (agent/request-error + CONTEXT_WINDOW_EXCEEDED → compact → retry), but bare HTTP 400 + exceed_context_size* is often classified as INVALID_REQUEST, so the recovery path never runs.

Reproduced on local 64K llama.cpp routes with Chinese-heavy agent sessions (heuristic far below window, server reports prompt over n_ctx).

Proposed core direction (decoupled)

Optional adapter capability, no engine names in core:

interface TokenCountResult {
  promptTokens: number
  source: 'provider' | 'tokenizer' | 'approx'
}

// LlmAdapter optional method — same assembly path as stream()
countPromptTokens?(options: GenerateOptions, signal?: AbortSignal): Promise<TokenCountResult>

• token-meter / compaction call through the runtime when present; on miss/timeout → keep heuristic.
• Adapters that expose HTTP tokenize/count (or native count APIs) opt in via route config, e.g. promptTokenCount: true | { endpoint, input, templateEndpoint }.
• Expand isContextWindowExceededError / stream classification so exceed_context_size* maps to CONTEXT_WINDOW_EXCEEDED before bare-400 → INVALID_REQUEST.

This stays provider-neutral: DeepSeek cloud, gateways, and local servers can each implement the method; core never hard-codes /tokenize or llama.cpp.

What the community can use today

While external PRs are closed, a stock-safe plugin implements the operational half without forking @deepseek-ai/*:

• llm/stream rewrite of overflow-like finish errors → CONTEXT_WINDOW_EXCEEDED
• agent/request-error fallback compact+retry for misclassified overflows
• HTTP tokenize against configured baseURL; when over thresholdRatio × contextWindow, synthesize CONTEXT_WINDOW_EXCEEDED for stock compact-retry

Install shape: npm / github: / link: bundle with dsh.bundle.patch, topic dsh-plugin.

Reference implementation:

https://github.com/tianyiming1/dsh-plugin-local-prompt-bridge

• Topic: dsh-plugin
• Install: pnpm add github:tianyiming1/dsh-plugin-local-prompt-bridge, then add @local/dsh-plugin-local-prompt-bridge to profile bundles

Verified on official DSH desktop path + 64K llama.cpp (Bonsai): tokenize max above threshold → synthesized CONTEXT_WINDOW_EXCEEDED → stock compact-retry recovered session (~56k → ~18k).

Ask

1. Is the optional countPromptTokens seam acceptable for a future core change?
2. Can overflow classification for exceed_context_size* land even sooner (small, high leverage)?
3. Until then, any objection to documenting the community plugin pattern for local OpenAI-compat tokenize?

Happy to split into: (a) overflow classifier only, (b) protocol + meter, (c) reference adapter opt-in — whenever external contributions reopen.

[tianyiming1]

中文摘要

问题： 官方 DSH 用固定字符/token 启发式估算上下文，中文长会话和工具结果常被低估。本地 llama.cpp 在真正超窗时返回 exceed_context_size_error，但 stock DSH 常把它当成普通 400，已有的 compact-retry 路径接不住。

社区方案（不改官方 core）： 独立 dsh-plugin，通过 llm/stream 钩子实现：

请求前调用本地服务 /tokenize 精算 token，超阈值主动触发压缩
将 exceed_context_size* 改写为 CONTEXT_WINDOW_EXCEEDED，走官方 compact-retry
参考实现： https://github.com/tianyiming1/dsh-plugin-local-prompt-bridge
（topic: dsh-plugin）

实测： 官方桌面版 DSH + 64K llama.cpp（Bonsai），token 约 5.6 万超阈值 → 自动压缩 → 回落约 1.8 万，会话可继续。

希望上游考虑：

可选 countPromptTokens 协议（provider 中立）
更早合入 exceed_context_size* → CONTEXT_WINDOW_EXCEEDED 分类

[tianyiming1]

Clarification / update (plugin defaults)

The linked reference plugin has been updated so defaults are provider-neutral:

• rewriteOverflow applies to all providers (no personal model/port hard-coded)
• routes: [] by default — proactive /tokenize is opt-in in the installer's own profile, aligned with their llm-pi-ai providers
• Bonsai / 18200 in the original post was only the verification machine, not a required default

Repo: https://github.com/tianyiming1/dsh-plugin-local-prompt-bridge (v0.1.1)

No change to the proposed core direction (countPromptTokens + better overflow classification).

[tianyiming1]

Clarification: plugin defaults are now provider-neutral (routes: [], global overflow rewrite). Bonsai/18200 was only the verification machine, not a required default. See v0.1.1 on the reference repo.

[tianyiming1]

Update (plugin v0.1.2): soft vs hard tokenize pressure

Follow-up from real usage on official DSH + local 64K llama.cpp (Bonsai2):

Bug we hit: synthesizing CONTEXT_WINDOW_EXCEEDED at a soft threshold (e.g. 0.45 × n_ctx) when the excess is mostly this-turn tool/file payloads:

1. Precise count ≈ 40k on a 64k window (still fits)
2. Soft line ≈ 29k → fake overflow → stock compact-retry
3. Compaction shrinks history, not current tool results
4. Still ≈ 40k > soft → fake overflow again → retries exhaust → turn fails

Plugin fix (v0.1.2):

Band	Behavior
≤ thresholdRatio (soft)	log only, allow dispatch
≤ hardRatio (default 0.90)	same — still fits real window
> hardRatio	synthesize CONTEXT_WINDOW_EXCEEDED for compact-retry
provider exceed_context_size*	still rewritten → CONTEXT_WINDOW_EXCEEDED

Repo: https://github.com/tianyiming1/dsh-plugin-local-prompt-bridge (commits 51d9389 / 0893cea)

Please treat soft vs hard pressure as part of any future core countPromptTokens design — soft fake-overflow loops are a footgun when tool payloads dominate. Draft updated in repo DISCUSSION.md.

[tianyiming1]

Update v0.1.3

Soft/hard refined further:

1. Soft: synthesize CONTEXT_WINDOW_EXCEEDED only when estimated removable history (messages before the latest user turn) is large enough that compact could land under soft. If the bulk is this-turn files/tools → allow (no fail-loop).
2. Hard: min(hardRatio × window, window − outputReserve) (default reserve 4096, or maxTokens if larger) so generation keeps budget.
3. Provider exceed_context_size* rewrite unchanged.

Repo: https://github.com/tianyiming1/dsh-plugin-local-prompt-bridge

[tianyiming1]

Update v0.1.4 — align with Cursor/Codex ~90% compact

After comparing production agents (Cursor ~90% auto-summarize, Codex model_auto_compact_token_limit capped ~90%, Claude Code auto-compact near the window):

Plugin change:

• Remove low soft-ratio fake-overflow / history heuristics
• Tokenize the outgoing prompt
• Compact only when promptTokens > min(thresholdRatio × window, window − outputReserve) (defaults 0.90 / 4096)
• Below that → allow dispatch (fixes the ~40k/64k false-fail)
• Over window / provider exceed_context_size* → still rewritten to CONTEXT_WINDOW_EXCEEDED for stock compact-retry

This is token-budget management (industry), not a client “remaining KV %” gauge.

Repo: https://github.com/tianyiming1/dsh-plugin-local-prompt-bridge
DISCUSSION.md updated accordingly.