You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
edit/write results give the model zero visibility into what actually changed — the diff exists in the log but never reaches the model
#336
I'm evaluating dsh as a serious coding-agent runtime and noticed something that seems like a blind spot worth discussing (I checked the source before posting, so this is grounded):
Observation. After a successful edit or write, the model receives only a one-line acknowledgement — The file has been updated successfully. (see formatEditOutput in packages/fs/tool-fs/src/edit.ts). The full before/after diff is computed (computeHunkDiffs, with context lines) and attached as presentationMeta — persisted to the session log and rendered as the UI diff card — but it is never rendered into the model's content. The README states this explicitly: "The full write or replacement text remains in the assistant tool-call arguments."
Why this is a real blind spot, not a nit:
replace_all: true can replace more (or fewer) occurrences than the model expected; the model has no signal that the actual mutation diverged from intent.
The diff basis is LF-normalized (before/after from the backend), so files with CRLF or odd whitespace can produce replacement behavior that surprises the model.
In general: the model knows what it said (its own args) but never what the world became. Its only self-verification path is a follow-up read or bash git diff — an extra round trip it must remember to make.
Why it should be cheap to fix: the diff is already in the session log (presentationMeta → meta.diffs). Rendering a bounded version into model content changes no invariant — "model-visible ⟺ logged" holds either way. The cost concern is real (tokens + append-only KV-cache growth until compaction), which is presumably why the Claude Code-style acknowledgement was chosen.
Options I'd love input on:
Include a bounded compact diff (e.g. N hunks / M lines, capped) in edit/write result content — maybe only when multiple hunks or replace_all was used.
A deployment-configurable toggle (per the "no hardcoded tunables" convention).
Keep results as-is and document a "verify-read contract" instead.
Bonus observation for the plugin ecosystem: the pipeline already makes this patchable from outside — at tools/post-execute, result.meta carries the persisted diffs (computed in createSuccessResult before the waterfall), and a listener can replace the result content. So a community plugin can give the model the diff today without any core change.
Is this working as intended, or is "the model never sees what changed" a known trade-off you'd like to revisit?
我在认真评估把 dsh 用作正式编码 Agent 运行时,注意到一个疑似盲区,想公开讨论一下(发帖前已核对源码,以下结论都有出处):
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
I'm evaluating dsh as a serious coding-agent runtime and noticed something that seems like a blind spot worth discussing (I checked the source before posting, so this is grounded):
Observation. After a successful edit or write, the model receives only a one-line acknowledgement — The file has been updated successfully. (see formatEditOutput in packages/fs/tool-fs/src/edit.ts). The full before/after diff is computed (computeHunkDiffs, with context lines) and attached as presentationMeta — persisted to the session log and rendered as the UI diff card — but it is never rendered into the model's content. The README states this explicitly: "The full write or replacement text remains in the assistant tool-call arguments."
Why this is a real blind spot, not a nit:
replace_all: true can replace more (or fewer) occurrences than the model expected; the model has no signal that the actual mutation diverged from intent.
The diff basis is LF-normalized (before/after from the backend), so files with CRLF or odd whitespace can produce replacement behavior that surprises the model.
In general: the model knows what it said (its own args) but never what the world became. Its only self-verification path is a follow-up read or bash git diff — an extra round trip it must remember to make.
Why it should be cheap to fix: the diff is already in the session log (presentationMeta → meta.diffs). Rendering a bounded version into model content changes no invariant — "model-visible ⟺ logged" holds either way. The cost concern is real (tokens + append-only KV-cache growth until compaction), which is presumably why the Claude Code-style acknowledgement was chosen.
Options I'd love input on:
Include a bounded compact diff (e.g. N hunks / M lines, capped) in edit/write result content — maybe only when multiple hunks or replace_all was used.
A deployment-configurable toggle (per the "no hardcoded tunables" convention).
Keep results as-is and document a "verify-read contract" instead.
Bonus observation for the plugin ecosystem: the pipeline already makes this patchable from outside — at tools/post-execute, result.meta carries the persisted diffs (computed in createSuccessResult before the waterfall), and a listener can replace the result content. So a community plugin can give the model the diff today without any core change.
Is this working as intended, or is "the model never sees what changed" a known trade-off you'd like to revisit?
我在认真评估把 dsh 用作正式编码 Agent 运行时,注意到一个疑似盲区,想公开讨论一下(发帖前已核对源码,以下结论都有出处):
现象。 edit 或 write 成功后,模型只收到一句确认——The file has been updated successfully.(见 packages/fs/tool-fs/src/edit.ts 的 formatEditOutput)。完整的 before/after diff 其实是被算出来的(computeHunkDiffs,带上下文行),作为 presentationMeta 挂上——落进会话日志、渲染成 UI 的 diff 卡片——但从不渲染进模型的内容。README 里写得很直白:"完整的新内容留在模型自己的工具调用参数里"。
为什么这是真盲区,不是吹毛求疵:
replace_all: true 可能替换了比模型预期更多(或更少)的位置;实际变更与意图偏离时,模型得不到任何信号;
diff 基底是 LF 归一化后的 before/after,遇到 CRLF 或怪异空白的文件,替换行为可能让模型意外,而它无从察觉;
一句话:模型知道自己说了什么(它自己的参数),却永远不知道世界变成了什么。它唯一的自证路径是事后补一次 read 或 bash git diff——一个它得记得去做、且多花一整轮的往返。
为什么修起来应该很便宜: diff 已经在会话日志里了(presentationMeta → meta.diffs)。把有上限的版本渲染进模型内容,不碰任何不变量——"模型可见 ⟺ 已记录"两边都成立。成本顾虑是真实的(token 消耗 + 追加式 KV 缓存到压缩前的持续计费),大概也正是当初选择 Claude Code 式一句话确认的原因。
想听大家意见的几个选项:
在 edit/write 结果内容里附带有上限的紧凑 diff(比如 N 个 hunk / M 行封顶)——或许只在多 hunk 或用了 replace_all 时附带;
做成部署可配置开关(符合项目"无硬编码可调项"的约定);
结果保持现状,改为文档化一个"改后自验(verify-read)契约"。
给插件生态的一个附赠观察: 这条管道从外部就能打补丁——在 tools/post-execute 时,result.meta 已携带持久化的 diffs(在瀑布之前由 createSuccessResult 算出),而监听器可以替换结果内容。也就是说,社区插件今天就能让模型看到 diff,完全不用改核心。
这属于"按设计如此",还是"模型永远看不到变更结果"这个取舍你们其实也想重新议一议?
All reactions