Repository navigation
Replies: 3 comments
|
Confirmed, and the mechanism is narrower than "the collector truncates": it trims at a byte offset with no UTF-8 boundary check.
while (this.bytes > this.maxBytes) {
const head = this.chunks[0] as Buffer
const excess = this.bytes - this.maxBytes
if (head.length <= excess) {
// Drop the whole head chunk (length ≥ 1 is guaranteed while over cap).
this.bytes -= head.length
...
} else {
// Trim the head so the retained window is byte-exact at the cap — a
this.chunks[0] = head.subarray(excess)
this.bytes -= excess
}
The "drop whole chunks from the head" wording in the class doc ( Two ways out, neither of which needs the cap to change: after I did not re-read the August audit you are arguing with, so I am not taking a position on what it concluded. The line above is what matters for the disagreement: |
|
@PerryLink — that is the half I could not get from the published package: I could see the line, not the intent written above it, and I would rather have your reading of it than mine. I am taking your coordinates ( One thing your reading leaves open, and it changes where a patch goes.
Isolated to the two branches, one string, same package (
Row 2 drops a whole chunk and still invents two characters. Row 3 drops a whole chunk and invents none. All three return This is not a synthetic corner, though it is narrower than "any real stream": With a single large write, the pipe slices at 65536 bytes, and 65536 is not a multiple of 3 — the second and third chunks begin on continuation bytes. Run that through the collector with The control matters as much as the finding: when each write is small enough to land in its own chunk — tens of thousands of them, the exact count depends on the pipe — no chunk begins or ends on a continuation byte. The retained window in that run still starts mid-character, but through the trim rather than through a chunk boundary; the two routes are independent. So the sentence I can defend is not "streams are chunked mid-character" but "when the write is large enough for the pipe to slice it, the slices are not character-aligned". Where that leaves your two invariants: choosing between them is the boundary half, and I am not asking you — or anyone — to choose here. What I am asking about survives either choice. At the moment the loop exits, one test on one byte — The same asymmetry decides where a fix would sit: applied only after One correction to my own post, since you read it closely enough for it to matter. Its snippet imports Two boundaries so we do not talk past each other. Adjacent ground, different ask: @TTTPOB's #9546 argues the cap should not touch what a program is handed at all. This thread asks only that a cut describe itself when it happens; that argument is theirs and I am not merging it into this one. And I am not re-opening the August audit either — you did not re-read it, and I am not asking you to. Authorship note: every reading here was taken on Reported by the OfferKuai Team — Founder: Zhaofeng (Yaming). Website: https://www.offerkuai.com/ | Contact: contact@offerkuai.com 中文版@PerryLink —— 这正是我从已发布包做不到的那一半：我能看见那一行，看不见写在它上头的意图；这一点我更想要你的读法，而不是我的。坐标采你的（ 你的读法留下一件事没定，而它决定补丁该落在哪。
把两条分支各自隔离出来，同一个字符串、同一个包（
第 2 行丢弃的是整块，照样凭空多出两个字符；第 3 行丢弃的也是整块，一个都没多。三行的返回值都是 这不是只在合成分块里成立，但也确实比"任何真实流都这样"窄： 单次大写入时管道按 65536 字节切，而 65536 不是 3 的整数倍 —— 第二、第三片都从续字节开始。把这股流传进收集器、 反过来的对照和这个发现一样重要：当每次写入都小到自成一片时 —— 分片数以万计，具体条数取决于管道 —— 没有任何一片以续字节开头或结尾。那一轮里保留窗口还是从字符中间开始，但那是裁剪造成的，不是分片边界造成的；两条路径彼此独立。所以我能站住的说法不是"流总是被切在字符中间"，而是"当写入大到需要管道切片时，切点不按字符对齐"。 那么你那两个不变量呢：在它们之间选一个，是边界那一半，我不在这里请你——也不请任何人——去选。我要问的那一半在两种选法下都成立。循环退出的那一刻，对一个字节做一次测试 —— 同一个不对称也决定了补丁该坐在哪：只加在 还有一处是我自己的稿子要改的，因为你读得足够细，细到会被它绊住。帖里的片段写的是 两条边界，免得各说各的。相邻但不同：@TTTPOB 的 #9546 主张上限根本不该碰交给程序的数据；本帖只主张"切了就说明白切成了什么样"。那是他的论证，我不往里并。八月的审计我也不重开 —— 你没重读，我也不请你重读。 声明：本回复里每个数字，都是把 npm 上发布的 报告：OfferKuai（Offer快）团队 —— 创始人 赵峰（Zhaofeng / Yaming）。官网：https://www.offerkuai.com/ ｜ 联系：contact@offerkuai.com |
|
You're right, and the correction matters for where the patch goes: a chunk boundary is not a character boundary, so if (head.length <= excess) {
// Drop the whole head chunk (length ≥ 1 is guaranteed while over cap).
this.chunks.shift()
this.bytes -= head.length
} else {
this.chunks[0] = head.subarray(excess) // :178
this.bytes -= excess
}If chunk 1 ends inside That narrows the fix further than what I wrote. Advancing to the next UTF-8 lead byte after The second option is simpler and keeps the byte-exact cap: store Thanks for the coordinates call-out — taking the built-file line numbers out of it was the right move, since those move with every bundle. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
A truncation is supposed to remove content. It is not supposed to alter what is left.
Here it alters.
When a subprocess collector keeps more output than it has room for, it drops the head of its buffer by bytes, with no regard for where a character ends. The first thing a reader sees can therefore be a character the command never printed — two replacement characters in place of the first Chinese word, in the reproduction below.
The timing is what makes this more than an off-by-one. The damage is created while the output is being buffered, not while it is being decoded, so every read path downstream inherits it — including the foreground read on the main path.
A deep audit posted in August concluded the opposite for that path. That one sentence is what this post disagrees with.
Every number below comes from running the published package,
@deepseek-ai/dsh-subprocess-local@0.2.1-rc.2(the currentnexttag;latestis older), not from reading the source. The reproduction is a script anyone can rerun.Where I agree with the earlier audit, and the one sentence where I do not
@Jiaaqiliu's audit, #3251, already covers two byte cuts of this shape —
utf8Tail()in the terminal session, and the subprocessreadFromdecoding each byte range on its own — and it already proposes the fix I would otherwise have proposed: step past10xxxxxxcontinuation bytes before decoding. That fix belongs to that report, and so does the principle behind it. I am not restating either as mine.One sentence in it does not hold:
It is safe from the failure that report describes. A single
readFrom(0)cannot split a character across two of its own increments. But it reads a buffer that was already cut, and the cut is exactly where a character got split. The foreground path is immune to the decoding fault and exposed to the buffering fault, and only the first of the two was checked.Where the character is invented
When output exceeds what the collector keeps, the collector drops the head of its buffer and holds the tail. It drops that head by bytes, at
this.chunks[0] = head.subarray(excess)— a raw slice. In the published package, that file contains no UTF-8 boundary handling at all.So the broken byte range lives in the buffer itself.
finalize(),snapshot()andreadFrom(0)all decode what is left, and by then the leading bytes are continuation bytes with no lead byte to belong to. They decode to U+FFFD. The rest of the text is intact and correct, which is what makes the result look trustworthy.Two characters were added. Neither result says anything about it.
How often this lands
Not a corner case. The outcome depends only on whether the cutoff falls inside a multi-byte sequence, and for non-ASCII output that is the usual outcome.
Randomized over outputs mixing CJK, Latin and emoji, counting only those longer than the cap:
A pure-ASCII control produced 0 of 500, and a known-positive control produced the replacement characters every time, so this is not a broken detector reporting zero.
Both controls earn their place. My first run reported zero hits out of two thousand and was wrong: the sample builder ended every output the same way, which pinned the cutoff to a character boundary every time. A measurement that cannot fail is not a measurement.
One run, two readers, two different contents
This part appears to be uncovered by the earlier reports.
The spill path appends the raw chunks, so the spill file is byte-exact. The in-memory tail is the part that was cut. Within a single run:
Same command, same bytes, two views that disagree, and nothing announces it.
truncatedistruein both cases — and it is the sametruewhether the head was lost on a clean boundary or through the middle of a character. A reader holding{ text, truncated: true }cannot tell which one they have.Two answers, one product
Same product, same stream, two implementations. Which one is right is not the question here — both are theirs.
Both halves of this are already written down in this repository, for the same stream.
The retention library carries its own purpose in the design record behind it,
.agents/notes/archived/architecture/2026-07-06-tool-result-retention-library.md:Its test list says what that has to hold up against:
The code is there as well. Two helpers, one for each direction a cut can go wrong. This post is about the second:
The same file handles the other kind of cut damage too, and names the reason it defends that case:
One module has therefore decided that a cut must not corrupt what it keeps, written both directions of the fix, tested all three codepoint widths at every cut point, and named the safety reason: damage that a validating reader would reject has to be prevented.
The subprocess collector is the other half. Its own comment describes the same operation:
The line that performs it does not consult a boundary:
Here is the part I am not claiming. The design record says migration is "deliberately separate follow-up work" (
:142). A tool that has not been migrated yet is a schedule, not a defect, and I am not filing a schedule.What I am filing is narrower, and it survives that sentence: while both implementations run, the field name is shared and the meaning is not. The record defines the flag in one sense only:
That is the definition a reader has, because it is the published one. Read against it,
truncated: truepromises shorter. On the subprocess path it can mean altered, and the same word covers both. A migration would end the divergence. Until it happens, the published meaning of a field describes one of the two implementations that set it.The standard this belongs to
This section is the only part of the post that carries an opinion. Everything above is measured.
The reason to write this up is not that one file is short a few lines of code. It is that a truncated tool output is consumed as an excerpt when it has quietly become evidence.
Almost everything an agent knows about the commands it ran arrives through this channel: build logs, test output, file paths, the last lines before a failure. Once that channel can alter content, the agent's account of what happened stops being verifiable — and the resulting failures will not look like truncation problems. They will look like a model misreading a build, or a reviewer trusting a log, or a summary that "loses" a detail nobody can find again.
I would put three requirements in a production agent's output contract, in increasing order of how much they cost:
const omitted = this.total - keptPrefix.length - keptSuffix.length— the count is taken after the trim, from the bytes that actually come back. The cap is an intention; the returned bytes are the fact.truncated: truemerges "the head was dropped cleanly" with "the head was dropped through a character". That is the difference between a shorter version of the output and a modified one, and no field on this path carries it.The first two are not proposals. They are this product's own standard, written down and implemented — in a different module, serving the same stream, by name. The third is the one nothing on this path can express.
So the disagreement is not about what the standard should be, and it is not about when the migration happens. It is about a standard the product set for itself, implemented in the module built for that stream, and absent from the code that stream actually runs through today.
This is the quiet failure, not the loud one
It needs separating from the reports that came before, because the consequence runs the other way.
@bings1230's #1449 and @TommeyChang's #8846 describe truncation that cuts a UTF-16 surrogate pair, leaving a lone surrogate: that text is not valid JSON, the request is rejected, and the session dies loudly with a 400. Credit for the boundary principle is theirs and @Jiaaqiliu's.
This one is quieter. A byte cut through a UTF-8 sequence yields U+FFFD, which is a valid character. It serializes, it is accepted, it renders. Nothing fails. The only symptom is text that says something the command did not say — and it passes every check the pipeline has.
The product's existing defence against cut damage is aimed at the loud case, and says so:
truncateWithoutSplittingSurrogatePairexists because "a strict JSON reader rejects a durable Session log" carrying an unpaired surrogate. That is a sound rule, and this failure goes around it rather than through it — a strict reader has no reason to reject U+FFFD, so there is nothing here for validation to catch.That is the reason to file it. The loud version gets fixed because it breaks something. This one breaks nothing.
What it implies once it accumulates (inference, not a reading)
Everything above is measured. This section is what those measurements imply, and it is labelled as inference.
Non-ASCII output is the ordinary case now — build logs, test output, paths, messages between agents. Each such character occupies 3–4 bytes, so any truncation of a non-ASCII stream lands on a boundary only by luck; measured, it landed inside a character 94–100% of the time.
Two consequences I expect to surface later rather than now:
What a fix costs
The boundary half already has a patch in that audit, and an equivalent helper already exists in the retention package. Applying either at the cut point costs at most 3 bytes of retained output — the bytes needed to reach the start of the next character.
The half still missing is the reporting side:
truncated: truecovers both and describes neither.The first is already the documented behaviour of the retention package. The second costs one field.
What this post does not ask for
Authorship note: both readings were taken on
0.2.1-rc.2— thenexttag at the time of writing — by executing the two packages as published on npm; no other version was run. The reproduction is in the post, so anyone can re-run it against the version they actually use and compare. Drafting assisted by AI; verification and publication by me. I have no engineering background — if any technical claim reads wrong, please call it out and I will re-verify against the toolchain and correct.Reported by the OfferKuai Team — Founder: Zhaofeng (Yaming). Website: https://www.offerkuai.com/ | Contact: contact@offerkuai.com
中文版
截断的职责是删掉内容，不是改动留下来的内容。
这一次它改动了。子进程收集器放不下那么多输出时，会把缓冲区头部按字节丢掉，不管字符到哪里结束。于是读者看到的第一眼，可能是命令从来没有打印过的字符 —— 下面的复现里，第一个中文词变成了两个替换字符。
时间点才是关键。损坏发生在输出被缓冲的时候，不是被解码的时候，所以下游每一条读取路径都继承了它，包括主路径上的前台读取。八月一份全仓审计对那条路径给出的结论正好相反，这一篇要商量的就是那一句。
下面每个数字都是跑出来的，不是在源码里读出来的 —— 跑的是 npm 已发布的
@deepseek-ai/dsh-subprocess-local@0.2.1-rc.2（当前next标签；latest比它旧）。复现是一段谁都能重跑的脚本。我与那份审计一致的地方，以及唯一不一致的一句
@Jiaaqiliu 的审计（#3251）已经覆盖了这个形状的两处字节切割：终端会话里的
utf8Tail()，以及 subprocess 的readFrom逐段独立解码。它也已经给出了我本来会提的那个修法 —— 解码前跳过
10xxxxxx连续字节。那条修法归那份报告，它背后的原则也归它。这两样我都不重述成自己的。
它里面有一句不成立：
对它描述的那种故障，这句话是对的：单次
readFrom(0)不会把一个字符劈在它自己的两次递增之间。但它读的是一个已经被切过的缓冲区，而那个切口正是字符被劈开的地方。
前台路径对解码故障免疫，对缓冲故障暴露。当时只查了前一种。
字符是在哪一步被造出来的
输出超过收集器能保留的量时，收集器丢掉缓冲区头部、留下尾部。它是按字节丢的，位置是
this.chunks[0] = head.subarray(excess)—— 一次裸切片。在已发布的包里，这个文件没有任何 UTF-8 边界处理。所以坏掉的字节区间就存在于缓冲区本身。
finalize()、snapshot()、readFrom(0)解码的都是剩下的部分，而此时开头那几个字节是连续字节、却没有前导字节可以归属。它们被解码成 U+FFFD。后面整段文本完好无损 —— 这恰恰是结果看起来可信的原因。两个字符被添了上去。两个结果里都没有一个字提到这件事。
多久撞上一次
不是边角情形。结果只取决于切口是否落在多字节序列内部，而对非 ASCII 输出来说，这就是常态。
在混合中日韩字符、拉丁字符与 emoji 的输出上随机取样，只统计超过上限的：
纯 ASCII 对照为 500 份中 0 份；已知必然命中的正对照每次都命中 —— 所以这不是"检测器坏了、报了个零"。
两个对照在这里都有用。我第一次跑出的结果是两千份里零命中，而那是错的：样本构造让每条输出的结尾都一样，把切口固定在了字符边界上。一个不可能失败的测量，不算测量。
同一次运行，两个读者，两份不同的内容
这一节是此前报告似乎没有覆盖的部分。
spill 路径写入的是原始数据块，所以 spill 文件字节精确。被裁掉的是内存里的尾部。于是同一次运行里：
同一条命令、同一批字节，两个视图不一致，而且没有声音。
truncated在两种情形下都是true—— 无论头部是在字符边界上干净地丢的，还是从字符中间丢的，都是同一个true。一个拿着{ text, truncated: true }的读者，分不出自己手上是哪一种。一个产品，两套答案
同一个产品，同一条流，两套做法。哪一套对，这里不需要判断 —— 两套都是他们自己写的。
这件事的两半，这个仓库都已经写在纸上了 —— 而且是为同一条流写的。
（下面几段官方原文是中文译述；行号指向原文，可以直接对照。）
保留器在自己的设计记录
.agents/notes/archived/architecture/2026-07-06-tool-result-retention-library.md里，是这么定位自己的：它的单测清单写明要扛住什么：
代码也确实在。两个函数，一个管切头的方向，一个管切尾的方向。本篇说的正是切尾这一个：
同一个文件对另一种切坏也做了处理，而且写明了它防的是哪一种后果：
也就是说，有一个模块已经想清楚了：裁剪不许污染它留下来的内容。 两个方向的修法都写成了具名函数，三种码点宽度都在每一个切点上测过，连安全理由都写出来了 —— 会让读者拒绝的那种损坏，必须防。
子进程收集器是另一半。它自己的注释，描述的是同一个动作：
真正动手的那一行，不看边界：
下面这段，是我不主张的。设计记录里明写：迁移是「刻意分开的后续工作」（
:142）。一个工具还没迁过去，那是排期，不是缺陷 —— 我不报排期。我要报的东西更窄，而且这句话正好挡不住它：两套实现同时在跑的时候，字段名是共用的，含义不是。
设计记录对这个标志只给了一层意思，也只有这一层：
一个读者手上只有这个定义 —— 因为这是公开发布的那个。按它读，
truncated: true承诺的是「短了一截」。而在子进程这条路径上，它可能是「被改过」，两种情形共用一个词。迁移做完，这个分叉就没了。在那之前，一个已发布字段的含义，只描述了设置它的两套实现里的一套。这属于哪一条标准
本节是全稿唯一带意见的部分。以上全部是实测。
要写这一篇，理由不是某个文件少了几行代码。理由是：一段被截断的工具输出，正在以"节选"的身份被消费，而它已经悄悄变成了"证据"。
一个 agent 对它跑过的命令所知道的一切，几乎都从这条通道进来 —— 构建日志、测试输出、文件路径、出错前的最后几行。
一旦这条通道会改动内容，agent 对"发生了什么"的叙述就不再可验证。
而由此产生的故障，看起来都不像截断问题。它们看起来会像：模型误读了构建，审阅者信了一份日志，或者一份摘要"漏掉"了某个细节、而那个细节再也找不回来。
我认为生产级 agent 的输出契约里该有三条要求，按代价从低到高：
const omitted = this.total - keptPrefix.length - keptSuffix.length—— 先 trim，再从真正交出去的字节里算。上限是意图，返回的字节是事实。truncated: true把"头部干净地丢了"和"头部从字符中间丢了"合成一个值。前者是输出的缩短版，后者是被改过的版本 —— 这个区别，这条路径上没有任何字段承载。前两条不是我的提案。它们是这个产品自己的标准，写下来了、也实现了 —— 在另一个模块里，服务的是同一条流，而且是点了名的那条。第三条，这条路径上没有任何东西能表达。
所以分歧不是"标准该定成什么样"，也不是"这次迁移值不值得做"。分歧是：一条产品为自己定的标准，写进了专为这条流建的那个模块，却不在那条流今天真正跑过的代码里。
这是安静的那种故障，不是响亮的那种
需要与此前的报告分开，因为后果方向相反。
@bings1230 的 #1449 与 @TommeyChang 的 #8846 描述的是切断 UTF-16 代理对、留下孤立代理项：那种文本不是合法 JSON，请求被拒绝，会话以 400 响亮地死掉。边界原则归它们，也归 @Jiaaqiliu。
这一条安静得多。按 UTF-8 字节切过去，产物是 U+FFFD，那是合法字符。它能序列化、能被接受、能渲染。什么都不会失败。唯一的症状是文本说了命令没说过的话 —— 而它能通过流水线上的每一道检查。
产品现有的那道防线，瞄的是响亮的那种，而且它自己也说了瞄的是哪种：
truncateWithoutSplittingSurrogatePair存在的理由，是"一个严格的 JSON 读取器会拒绝携带孤立代理项的持久会话日志"。这条规矩没错。而这一种故障是绕过它，不是穿不过它 —— 严格读取器没有任何理由拒绝 U+FFFD，所以这里没有东西留给校验去抓。这正是要报它的理由：响亮的那种因为会坏而被修掉。这一种什么都不坏。
它攒起来之后会怎样（推断，不是读数）
以上全部是实测。本节是这些读数所蕴含的东西，已标明为推断。
非 ASCII 输出如今是常态 —— 构建日志、测试输出、路径、agent 之间的来回。每个这类字符占 3–4 字节，所以任何一次对非 ASCII 流的截断，只有靠运气才落在边界上：实测中，94–100% 的情况没有落在。
两个我预期"以后才浮上来"的后果：
修一下要付多少
边界那一半，那份审计里已有补丁，保留包里也已有等价的具名函数。在切口处应用任一种，代价是保留输出最多少 3 个字节 —— 就是走到下一个字符起点的距离。
还缺的是报告那一半：
truncated: true同时覆盖两者、又都不说明。第一条已经是保留包写明的行为。第二条的代价是一个字段。
本帖不要求什么
声明：两处读数都取自
0.2.1-rc.2（写作时next标签指向的版本），在 npm 已发布的两个包上直接执行所得；没有跑过别的版本。复现脚本就在帖子里 —— 任何人在自己实际使用的版本上都能重跑一遍核对。文稿撰写由 AI 协助；核验与发布由我本人负责。我非工程背景 —— 若任何技术表述有误，请直接指出，我会对照工具链重新核实并更正。本报告由 OfferKuai（Offer快）团队 提交 —— 创始人：Zhaofeng（Yaming）。官网：https://www.offerkuai.com/ | 联系：contact@offerkuai.com
All reactions