[Bug] Subprocess spill files can silently lose bytes after short writes

[GCSimba]

I reproduced an output-integrity issue on current master at 687f282e8625c6b8d11ff58211a90524be2e4317 (0.2.1-rc.2): a subprocess spill file can silently lose bytes after a successful partial fs.writeSync() call, while still being advertised as full output.

Trigger and impact

When a subprocess stream exceeds its in-memory limit, OutputCollector.spillAll() replays buffered chunks and appends the current chunk to a spill file. Both write sites ignore the byte count returned by writeSync(). If that count is positive but smaller than the buffer length, the remaining bytes are never written. No exception reaches the existing failure handler, so the incomplete file keeps its advertised spill path.

The bounded memory tail can remain correct while the file intended to recover full command output is incomplete. Users or agents consulting that file can therefore receive silently truncated content. This concerns exceptional storage conditions; it is not a claim that ordinary file writes routinely return short counts.

Reproduction

The regression uses the real collector and real temporary files. It intercepts only writes to the collector's own descriptor, makes selected calls actually write and return one byte, and delegates other I/O to Node. With a four-byte memory limit, pushing AAAA, BBBB, and CCCC should produce AAAABBBBCCCC on disk.

Short-write location	Actual file before the fix
Initial buffered replay	ABBBBCCCC
Overflow-triggering chunk	AAAABCCCC
Later append	AAAABBBBC
Every multi-byte write	ABC

The original implementation exposes these incomplete files without invoking its failure reporter. A final short write also returns without attempting the remainder, so a subsequent storage error is not observed during that append. Five of the initial seven regression cases fail against the original implementation.

The final regression file contains ten tests covering ordinary writes, UTF-8 bytes, all three write positions, repeated partial progress, and ENOSPC after partial progress. The injection intercepts native fs.writeSync, including calls made inside Node's writeFileSync, rather than mocking completion away.

Minimal patch

Use writeFileSync(fd, Buffer) at both append sites. Node completes partial writes on the existing descriptor without reopening, truncating, or closing it. If completion throws, the existing handler discards the incomplete spill, withdraws its path, reports once, and continues collecting the bounded memory tail.

• Standalone core commit: one production module, regression coverage, and paired documentation.
• Branch / validated branch commit.

The existing exception-injection test is updated to intercept the new write operation. Public API, dependencies, configuration, output format, caps, and file lifetime are unchanged.

The branch also contains three independent validation fixes, each in its own commit: honor the CLI help concurrency override, avoid treating a uv artifact byte count as a commit reference, and use a directory junction for the Windows image-containment fixture. The core commit can be taken independently.

Validation

Node 24.19.0, pnpm 11.28.5; Windows plus Ubuntu on a real WSL2 Linux kernel and ext4 filesystem.

• Linux: five owning test files, 205 tests pass, including all 95 existing spawn tests. output.ts has 100% statements, branches, functions, and lines under the normal repository configuration.
• Windows: the ten new tests pass under the normal configuration; the existing POSIX spawn suite is covered by the Linux run.
• Built-export smokes on both platforms use a real Node child, stdout and stderr, UTF-8 content, and 274 injected short writes per run. Both spill files remain byte-complete.
• The assembled bash-spill recorded-session replay passes on Linux through the shipped headless profile using built artifacts. No expected snapshots were changed.
• Full Linux build: all nine artifact/type-check steps pass. Windows aggregate type checks also pass.
• Full lint, including prerequisite compilation: zero errors; one pre-existing unused-disable warning in the experimental CoT translation plugin.
• Documentation: 42 of 43 aggregate checks pass directly. The graph generator exhausts the Linux runtime's default approximately 2 GiB heap; rerunning it with a 4 GiB heap passes and confirms all seven graph documents are current. The original aggregate exits nonzero; every component now has passing evidence. No generator or expected content was changed to obtain that result.

Commands used for the Linux owning tests and repository checks:

node node_modules/vitest/vitest.mjs run \
  packages/subprocess/subprocess-local/tests/output.spec.ts \
  packages/subprocess/subprocess-local/tests/spawn.spec.ts \
  scripts/gen-cli-help.spec.ts \
  scripts/project-doc-site.spec.ts \
  scripts/verify-repository-references.spec.ts \
  --maxWorkers=2 --coverage \
  --coverage.include=packages/subprocess/subprocess-local/src/output.ts
DSH_GATE_CONCURRENCY=1 corepack pnpm run build
DSH_EXAMPLE_MODE=lib DSH_SNAPSHOT_MAX_WORKERS=1 DSH_SNAPSHOT_MAX_CONCURRENCY=1 \
  corepack pnpm run test:snapshot snapshots/session/headless.snapshot.ts -t bash-spill --reporter=dot
DSH_GATE_CONCURRENCY=1 corepack pnpm run doc-sync
NODE_OPTIONS=--max-old-space-size=4096 corepack pnpm run verify-doc-graphs
NODE_OPTIONS=--max-old-space-size=4096 DSH_GATE_CONCURRENCY=1 DSH_OXLINT_THREADS=2 \
  corepack pnpm run lint

Related reports and limits

Before publishing, I searched public Discussions, issues/PRs, and commits for short writes, writeSync, and OutputCollector; I did not locate an equivalent short-return-value report. #260 and #5175 concern thrown spill I/O failures, while #7854 concerns the spill size cap. This report addresses a separate remaining failure mode in the same subsystem.

Fault injection establishes the handling defect and regression behavior. It does not measure real disk-full/quota frequency or prove crash/power-loss durability. Not every shell/backend combination has been exercised, and macOS was not tested dynamically. Node 22.19.0's built-in implementation was checked at source level; dynamic verification used Node 24.19.0.

AI-assisted audit, reproduction, implementation, and validation. The commits include a Codex co-author.

[PerryLink]

Both write sites are exactly where you say, and the failure is real. Confirmed on 687f282e86 (0.2.1-rc.2):

• packages/subprocess/subprocess-local/src/output.ts:211 — for (const prior of this.chunks) writeSync(fd, prior)
• :213 — writeSync(this.spillFd, chunk)

Neither reads the return value, and nothing else in spillAll() compares written bytes against the buffer length, so a short count is indistinguishable from success. The surrounding try/catch (:214-226) only sees throws.

Two details that make the impact a bit worse than "the file is short":

1. The path is published at :209 (this.spillFile = file) before either write runs. So the moment the first write returns short, the collector is already advertising a spill path whose contents are incomplete. The only thing that clears it is discardSpill(), and discardSpill() is reached from exactly two places: the catch at :218, and the whole-stream cap check at :192-194. A short write hits neither.

2. The in-memory tail can therefore look perfectly healthy while the recovery artifact is truncated — which is the worst combination for an agent that decides "the tail was cut off, go read the full output file".

Your four-byte arithmetic checks out, by the way. With AAAA/BBBB/CCCC and a short write only on the initial replay, you get A + BBBB + CCCC = ABBBBCCCC; short-writing the chunk paths instead gives ABC. Both match your table.

On the fix shape, two things I would add to what you propose:

• A loop over writeSync(fd, buf, offset, length) that advances until the buffer is exhausted is the correct primitive, but it needs a zero-progress guard. writeSync returning 0 for a non-empty buffer would spin forever otherwise; treat 0 as a failure and route it to the same discardSpill() + onFailure path you already have.
• Whatever the loop does, the "short write" outcome should end in discardSpill(), not in "leave a partial file and keep advertising it". That is exactly the policy the code already applies to open failures and to the whole-stream overrun at :192-194: an incomplete artifact is not an advertised artifact. Reusing that exit keeps the contract single.

One framing note, since you were careful about it in your report and it is worth keeping: this is not an ordinary-write complaint. write(2) may return a short count on ENOSPC/quota exhaustion, on a signal-interrupted write, and on Windows partial WriteFile completion; for regular files under normal conditions it returns the full length. So the trigger is exceptional, but the consequence — silent truncation of the one artifact meant to hold the full output — is a correctness issue, not a cosmetic one.