DSH | dsh-context-budget | Keep a local model's context at a size your GPU handles well

[d3vmeh]

Project URL:
https://github.com/d3vmeh/dsh-context-budget
npm: https://www.npmjs.com/package/dsh-context-budget

Introduction:

DSH compacts a conversation when it reaches a fixed fraction of the model's declared context window. On local hardware the declared window is not very practical: my Qwen3.8-27B advertises 256K, but on a Strix Halo it reads prompts at ~300 tokens/s when the context is small and ~70 tokens/s past 100K. A turn that started in seconds early in the session starts in half an hour later on, and the GPU hung near 170K before the default 80% threshold was reached.

dsh-context-budget measures how expensive the context has become and warns, or compacts early, before that point. Before every agent step it runs up to three checks per provider route:

• a hard ceiling in tokens (the one that prevents the crash),
• observed slowness: the last reply's time to first token exceeded your limit,
• predicted slowness: context size divided by the measured prefill rate exceeds your limit (the wait if the server's prompt cache were lost).

Any configured check can trip; unset ones are skipped. Default action is warn (one line in the DSH terminal); compact asks DSH's own compaction engine to summarize everything except the most recent retainTokens. A /context-budget command shows the live numbers, including an estimate of what compacting now would cost, because on local hardware a compaction is not free (the summary prompt and the rewritten context both prefill cold; on my machine about 25 to 35 minutes each at 120K).

How it integrates with DSH:

A small cordis plugin. It measures on the llm/stream waterfall (time to first chunk plus the uncached prompt tokens from DSH's usage report, so it works with any provider and never talks to the model server), decides on agent/pre-step using ctx.tokenMeter, and compacts through the agent's own compaction engine (compactRegion) with the same range selection compaction-basic uses. It ships a dsh.bundle patch, so installation is:

dsh plugin --profile web add dsh-context-budget

then in ~/.dsh/profiles/web/cordis.patch.yml:

- id: context-budget
  config:
    providers:
      llamacpp:
        hardCeilingTokens: 110000
        maxColdPrefillMs: 600000
        retainTokens: 24000
        action: warn

What it looks like in use:

context-budget: llamacpp session=059a680f ceiling: 10825 tokens > ceiling 5000 (10825 tokens, rate 232 tok/s, compacting now ~2.5 min)

context-budget (llamacpp / qwen3.8-q4-long)
  context: 10891 tokens   ceiling 5000   [TRIP]
  measured rate: 232 tok/s (1 sample)
  predicted cold: 0.8 min   limit 10.0 min
  compact now: ~2.5 min   (retain 24000)
  action: warn

If you only want the static ceiling, you do not need this plugin: compaction-basic's modelPolicies with a per-model thresholdRatio does that today. This plugin adds the measured checks and the cost estimate.

Companion: dsh-llm-gate (#4995) keeps the number of requests within your server's slots; this one keeps each request's size within what the GPU handles well.

MIT License. Feedback welcome!

> Unofficial project, independently developed and maintained by community members.

[liyangbing]

建议把三个预算边界分开显示

这个插件的 measured/predicted slowness 对本地模型很有帮助。需要特别避免把几个不同数字混成一个“context size”：provider usage 是一次请求的计量，surface estimate 是模型可见内容的估算，compaction cost 则是额外的 summary/rewrite 请求成本；提前 compact 可能降低后续压力，但不是免费操作，也不会抹掉已经发生的调用成本。

建议在集成和 UI 中固定这些证据：
1. 记录 provider、model、tokenizer/估算方法、缓存命中状态、样本时间和 measured prefill rate，区分“实际 usage”与“预测 cold prefill”。
2. 把 hard ceiling、observed TTFT、predicted cold-prefill 和 compaction threshold 分成独立的触发原因；一个触发不应覆盖其他原因。
3. compaction 前后记录选中的 range、retainTokens、summary 输入/输出、成功提交或回滚状态；失败时不能把部分结果当成已压缩。
4. action=warn 与 action=compact 都要有 bounded wait、取消和并发 guard，避免慢模型上重复触发多个 compaction。
5. 所有状态按 profile/session/provider 隔离，且提示词、工具结果、凭据和本地路径不能进入面向用户的诊断日志。

手册中有一页独立的 token accounting 说明，专门区分 surface、provider usage、compaction 和 retry 计量，并给出长 Session 的核对方法：Interpret token estimates, provider usage, and compaction pressure。它不是官方 DeepSeek 文档，也不代表对该插件的兼容性认证；请在目标 DSH revision、模型和硬件上重新验证。

[d3vmeh]

I added the per-sample detail in the new version (0.2.0). Now each rate sample records when it was measured and whether the server answered from a warm prompt cache. the /context-budget command lists them:

  measured rate: 68 tok/s (3 samples)
    3 min ago   8400 tok in 37 s   cold
    12 min ago   2100 tok in 8 s   warm
    41 min ago   12800 tok in 55 s   cold

Warm samples have flattering timings, so seeing the mix next to the average makes the rate easier to trust. Update with: dsh plugin --profile web add dsh-context-budget@0.2.0 (or bump the version in your profile).

[d3vmeh]

Thanks, that's a useful way to think about it, and most of it is already how the plugin works. The three checks are evaluated on their own and the log line names whichever one tripped, one line per step, so none of them hides another. When the plugin does trigger a compaction it logs the result either way, and a failed compaction is never treated as done. A slow reply is only remembered for the context it was measured on; once a compaction rewrites that context it's forgotten, so one compaction can't set off the next, and the summary requests themselves aren't measured. dsh's own engine also refuses to run two compactions at once. State is kept per provider and per session, and the logs only ever contain counts and timings, never prompt text, tool output, or file paths.

The part that isn't there yet is per-sample detail. /context-budget shows the measured rate and how many samples it rests on, and keeps real usage apart from the predicted cold prefill, but it doesn't record when each sample was taken or whether the cache was warm. That's something I can add in the next version.