[Bug] TextRetainer drops U+FEFF at window starts and underreports omitted bytes

[GCSimba]

Summary

TextRetainer can silently remove a complete U+FEFF codepoint from retained text. In a truncated result, the reported exact omission count then understates the bytes absent from the returned text. In an untruncated result, content can disappear while truncated remains false.

Independently reproduced against master at d743267388641bc76f17c45ce8b4c231aed1d32c (0.2.1-alpha.2). Master was rechecked before posting and still points to that commit. The patch is available as a single fork commit; no upstream PR is opened, in accordance with the current contribution policy.

Minimal reproduction

With Node 24.19.0, the following library-level example can be evaluated as ESM from the checkout root on the baseline above:

import { TextRetainer } from './packages/util/output-retention/src/index.ts'

const input = 'abc\uFEFFZ'
const retainer = new TextRetainer({ kind: 'tail', maxBytes: 4 })
retainer.push(input)
const result = retainer.finish()
const inputBytes = new TextEncoder().encode(input).length
const retainedBytes = new TextEncoder().encode(result.text).length

console.log({
  retainedCodePoints: [...result.text].map(char =>
    `U+${char.codePointAt(0).toString(16).toUpperCase().padStart(4, '0')}`),
  inputBytes,
  retainedBytes,
  actualMissingBytes: inputBytes - retainedBytes,
  truncated: result.truncated,
  omittedBytes: result.omittedBytes,
})

The U+FEFF is inside the original stream, not at its beginning. Its three UTF-8 bytes and the final Z fit the four-byte tail window exactly.

Field	Baseline output	Expected / patched output
Retained codepoints	U+005A	U+FEFF, U+005A
Input bytes	7	7
Returned UTF-8 bytes	1	4
Bytes actually missing	6	3
omittedBytes	{ kind: 'exact', count: 3 }	{ kind: 'exact', count: 3 }
truncated	true	true

All three strategies also return Z for a complete \uFEFFZ input that fits the budget, while reporting omittedBytes: { kind: 'none' } and truncated: false.

Root cause and observable impact

The shared decoder uses new TextDecoder(). Its default BOM handling removes U+FEFF at a decode start. Node's TextDecoder documentation confirms that ignoreBOM: true preserves the mark in decoded output.

finish() decodes retained prefix/suffix windows independently when a middle gap exists, and decodes the complete buffer when nothing is omitted. A window start is not necessarily a file header. The omission count uses retained raw byte lengths after partial-codepoint trimming, so the decoder's additional three-byte deletion is not counted.

This reaches a real product path: previewLine() applies head retention to every displayed grep match, even when the line fits the budget. A real UTF-8 file containing the JavaScript string 'ordinary line\n\uFEFFtarget\n', searched through the grep tool with { pattern: 'target', path: 'feff.txt' }, returns Line 2: target rather than Line 2: \uFEFFtarget. The search metadata loses the same character and does not report line truncation. The U+FEFF is on the second line, not a file-level BOM. The source file itself is not modified.

The same helper is also used by job output retention and Session-reference retention. Those callers share the decoding behavior; the new assembled regression specifically covers grep.

Minimal patch

-const decoder = new TextDecoder()
+const decoder = new TextDecoder('utf-8', { ignoreBOM: true })

The production behavior change is confined to this decoder option. Existing partial UTF-8 cuts are still discarded, and malformed interior bytes retain non-fatal U+FFFD replacement. Callers that need file-header BOM removal must apply that file-format policy before retention.

• Immutable commit: cbbd6cd0b50a2118e5333579143dae546b865bb4
• Branch: codex/fix-retained-utf8-bom
• Owning regressions: all strategies, adjacent and separated windows, split byte chunks, partial U+FEFF cuts, and malformed-byte handling.
• Real packaged-ripgrep regression: matched interior-line U+FEFF preservation.
• Recorded headless scenario: real shipped dsh --profile headless, filesystem, ripgrep, tool pipeline and Session persistence; only the model response stream is authored/replayed.

The commit changes nine files, including the bilingual README pair and pairing record. There are no public TypeScript API, package-export, dependency, configuration, or Session-format changes.

Verification

Node 24.19.0 and the repository-declared pnpm 11.28.5 were used, with frozen-lockfile installs. The following are actual local results, not a claim about remote CI:

Command / scope	Result
pnpm exec vitest run packages/util/output-retention/tests/output-retention.spec.ts --coverage --coverage.include='packages/util/output-retention/src/index.ts'	Windows and Linux: 50 passed; statements, branches, functions and lines each 100%.
pnpm exec vitest run packages/fs/tool-fs-search/tests/integration.spec.ts packages/fs/tool-fs-search/tests/tools.spec.ts packages/context/session-reference/tests packages/jobs/tool-jobs/tests	Linux: 238 passed, no skips. Windows: 237 passed; one existing POSIX-filename case skipped.
pnpm run test:snapshot snapshots/session/headless.snapshot.ts -t fs-grep-feff	Linux source-mode profile: selected scenario passed; complete persisted text/metadata compared and workspace unchanged.
pnpm run test:snapshot scripts/session-snapshot-corpus.corpus.ts	Three corpus checks passed on both platforms.
pnpm run test:docs	21 gates passed.
pnpm run lint:contracts-ready	Final full lint passed with one existing unused-disable warning.
pnpm run build:lib:client	Passed.
pnpm run build:native-system	Passed on Windows; the Linux native addon was also built locally.
Built ESM output-retention/lib/index.js smoke	Preserves \uFEFFZ with exactly three omitted bytes.
Normal commit/push hooks	Staged pairing, lint, whitespace and vendor guard passed; pre-push pnpm run typecheck passed, including the Host library build and Client contract typecheck.

On the unpatched module, eight of the ten added library cases failed at the intended U+FEFF assertions; the two partial-codepoint controls passed. The real grep regression also failed before the fix. After generating the final POSIX Session fixture, temporarily reverting only the decoder option made the assembled replay fail solely because U+FEFF disappeared from tool-result text and search metadata. Restoring the option made that same read-only replay pass.

pnpm run doc-sync initially finished with 41 passed and two failed on Windows. The CLI-help generator killed a web --help child at its deadline during concurrent checks; pnpm run verify-cli-help passed separately with all seven help pages current. The documentation-site test failure was Windows EPERM when creating a file symlink; the identical seven-file documentation-site test command passed all 151 tests on Linux. The Windows doc-sync aggregate was not rerun and is not represented as an all-green run. No system symlink permissions were changed.

Scope and limits

• The final Session scenario reuses the existing POSIX fs-search composition and was regenerated/replayed on Linux. Native Windows library and real-file grep tests passed; this is not a Windows assembled-Session snapshot claim.
• Linux source-profile runs reused platform-neutral generated typert.host.js contributors from the current Host build, with a locally built Linux native addon.
• No shared normalizer was changed, and no committed historical Session generation was edited.
• The full repository coverage suite, full snapshot suite, real-model API tests and Windows Wine lane were not run. No remote workflow run was reported for the fork commit.
• This patch preserves retained text. It does not change file decoding, filename BOM policy, or write behavior.

Before posting, public Discussions, issues/PRs, commit messages and indexed fork code were searched for the relevant TextRetainer/BOM/U+FEFF/ignoreBOM terms. No equivalent earlier report or patch from another contributor was found within that search. #4407 concerns raw ripgrep stdout overflow, not this decoding/omission mismatch; the indexed fork match concerns raw-rg decoding rather than the shared retainer. This does not establish that nobody has discovered it privately or on an unindexed branch.

AI-assisted repository audit and implementation. The local repro and verification results above were captured during that work; the patch includes Co-authored-by: Codex <noreply@openai.com>.

[PerryLink]

这条完全成立，而且根因、触发点、计数错误三处我都逐行核对过 —— 与你的描述一字不差。补充三样：为什么产品路径上必然命中、同缺陷的另外三个消费者、以及仓库里已有的修法先例。

1. 根因就在你指的那一行

packages/util/output-retention/src/index.ts:201：

const decoder = new TextDecoder() // utf-8, non-fatal: internal malformed bytes → U+FFFD

（上一行 :200 是 const encoder = new TextEncoder()。）缺 { ignoreBOM: true }，与你的补丁完全一致。

2. 触发点：有预算省略时前后缀各自独立 decode

index.ts:368-370：

const text = budgetOmitted > 0
  ? decoder.decode(keptPrefix) + decoder.decode(keptSuffix)
  : decoder.decode(concat([prefix, suffix]))

⇒ 后缀窗口的起点位于流中间，但解码器仍把它当作"解码起点" ⇒ 该处的 U+FEFF 被当作 BOM 剥掉。这正是你说的"window starts"。

3. 少报的机制

index.ts:372-377：

// Report omission against the bytes ACTUALLY returned…
const omitted = this.total - keptPrefix.length - keptSuffix.length
const truncated = omitted > 0

计数基于裁剪后的原始字节，而 decoder 额外吞掉的 3 字节不进入这个减法 ⇒ 正是"omittedBytes 少报 3 字节"。

而 budgetOmitted === 0（没触发省略）时走 :370 的整段 decode，此时 omitted = 0 ⇒ truncated: false、omittedBytes: { kind: 'none' }，内容却少了一个 codepoint —— 你的第二个断言也成立。这一条比少报更隐蔽：它连 truncated 都不置位。

4. ⭐ 为什么产品路径上"必然"会命中

packages/fs/tool-fs-search/src/search-core.ts:326：

const retainer = new TextRetainer({ kind: 'head', maxBytes })

它在 previewLine() 里（:325 起）—— 也就是说每一条显示出来的 grep 命中都要过一遍 head 保留，即使整行本来放得下。所以行内（非文件头）的 U+FEFF 会从工具结果与搜索元数据里消失，而且不报 line truncation。

这一点建议写进报告的"影响面" —— 它把"一个 util 的边界条件"变成"每次 grep 都可能静默改字符"。

同缺陷还影响另外三处消费者：

• packages/jobs/tool-jobs/src/index.ts:88（tail）与 :94（head）
• packages/context/session-reference/src/projection.ts:162（headTail）
• packages/experimental/tool-terminal/src/render.ts:57

5. ⭐ 修法与本仓既有做法一致（这条最有利于被采纳）

全仓 ignoreBOM 只有 3 处命中，output-retention 不在其中：

• packages/api/terminal-controller/src/terminal.ts:183 → const decoder = new TextDecoder('utf-8', { ignoreBOM: true })
• packages/ptc-runtime/ptc-runtime-node/src/index.ts:253-254 → stdout / stderr 两个 decoder 同样带 ignoreBOM: true

⇒ 所以你的补丁不是"新增一种做法"，而是"照仓库自己已有的写法补齐一处遗漏"。建议在报告里直接引这两个位置。

6. 你那条限定是对的，我确认一下

你写"callers that need file-header BOM removal must apply that file-format policy before retention" —— 正确，而且现状下没有任何 caller 在 retainer 之前做文件头 BOM 策略 ⇒ 补丁不会破坏任何现有 BOM 语义。这句可以放心保留。

顺带排一个容易混淆的点（避免别人拿它反驳你）：packages/client/ui-sidebar-documentpreview/src/client/markdown/frontmatter.ts:11 的 const FRONTMATTER = /^\uFEFF?---…/u 里的 \uFEFF 是有意允许文件头 BOM 的 file-format 策略，与 retainer 无关，也不受本补丁影响。如果有人在 review 里拿它出来，用这句回。

7. 版本

你标 0.2.1-alpha.2 / commit d743267388641bc76f17c45ce8b4c231aed1d32c —— 我复核了 git rev-list --count d743267388..origin/master = 0，你标的就是当前 master，未过期。

8. 用户侧 workaround

在走到 retainer 之前自己剥/记 BOM；或对含 U+FEFF 的行改用 read 而不是 grep 预览。