[fs-search] grep/glob hard-fail with SEARCH_RAW_OUTPUT_OVERFLOW once raw rg --json stdout exceeds the seam cap — spill-backed full parse

[edisonzerolam]

Title: [fs-search] grep/glob hard-fail with SEARCH_RAW_OUTPUT_OVERFLOW once raw rg --json stdout exceeds the seam cap — spill-backed full parse

Labels: bug

Summary

grep / glob throw SEARCH_RAW_OUTPUT_OVERFLOW whenever raw ripgrep stdout exceeds rawOutputMaxBytes (default 20 MB). The presentation layer already bounds what the model sees (first 250 matches × per-line previews), so truncation was clearly the intended contract — but the transport fails first. Any sufficiently broad search becomes a dead end, and the documented full-result spill (ctx.spillStore.saveText) never gets a chance to run.

Why this is easy to hit

• Broad alternations over dependency-heavy trees (node_modules, vendored bundles).
• A single long minified line suffices: each JSON match record embeds the whole line text. Verified with the packaged rg binary: one 31 MB line produces a ~63 MB --json stream, and --max-columns=4000 does not change it (max-columns does not apply to JSON output).

Root cause

completeStdout() throws whenever the seam reports a lossy read. Meanwhile the subprocess seam already supports complete-stream capture via collect-spill dispositions — stdout: { maxBytes, spill: { maxBytes } } — which the bash/pwsh tools already use. fs-search just doesn't opt in.

Proposed fix (~40 lines)

1. Spawn rg with stdout: { maxBytes: rawOutputMaxBytes, spill: { maxBytes: RAW_OUTPUT_SPILL_MAX_BYTES } } (budget 2e8).
2. completeStdout(): on a lossy read with spillPath, return { spillPath } instead of throwing; keep the hard error only when nothing recoverable exists or the stream exceeds even the spill budget.
3. New rawRunLines() async generator: within the in-memory window split the retained text; overflowed runs stream-parse the spill file line-by-line and unlink it when iteration ends.
4. Both tool loops iterate rawRunLines(run.stdout); grep parses records per line.

Result: the hard failure becomes "parse everything → let the existing inline caps bound the page", exactly mirroring the existing >250-match cap path. Streams beyond the spill budget still fail as today.

Verified red/green with a harness that mirrors the seam's tail-window/spill semantics plus the real packaged rg binary: pre-patch reproduces the throw on a single 31 MB-line fixture; post-patch returns matches with truncated-line preview markers, spill temps are cleaned up, and small-region behavior is unchanged.

Patch against lib/index.js (packages/fs/tool-fs-search, @deepseek-ai/dsh-tool-fs-search 0.1.1-rc.2)

--- a/lib/index.js
+++ b/lib/index.js
@@ -2,7 +2,9 @@
 import { MAX_TIMER_DELAY_MS } from "@deepseek-ai/dsh-timeout";
 import { isAbsolute, relative, sep } from "node:path";
 import { defineTool } from "@deepseek-ai/dsh-tools";
-import { existsSync } from "node:fs";
+import { createReadStream, existsSync } from "node:fs";
+import { unlink } from "node:fs/promises";
+import { createInterface } from "node:readline";
 import { HarnessError } from "@deepseek-ai/dsh-llm";
 import { ItemRetainer, TextRetainer } from "@deepseek-ai/dsh-output-retention";
 //#region lib/types/search-core.js
@@ -19,10 +21,10 @@
 * install is required, and no shell layer exists between the argv vector and
 * ripgrep, so no shell quoting is involved. Raw `rg` stdout is an internal
 * transport detail: the tools request a per-run stdout capture budget from the
-* subprocess seam, parse only complete in-memory stdout within
-* `rawOutputMaxBytes`, and never read spill files. The model-facing recovery
-* artifact is the formatted result saved through `ctx.spillStore.saveText()`
-* ({@link trySaveFormattedResult}).
+* subprocess seam and parse the complete raw stdout - in memory within
+* `rawOutputMaxBytes`, else streamed from the seam's spill capture of the
+* full stream. The model-facing recovery artifact is the formatted result
+* saved through `ctx.spillStore.saveText()` ({@link trySaveFormattedResult}).
 *
 * @module @deepseek-ai/dsh-tool-fs-search/search-core
 */
@@ -31,6 +33,14 @@
 * `rawOutputMaxBytes` config), matching Claude Code's ripgrep raw buffer.
 */
 const RAW_OUTPUT_MAX_BYTES = 2e7;
+/**
+* Cap in bytes on the seam spill file capturing the COMPLETE raw stdout once a
+* run overflows the in-memory window. An overflowing search no longer fails
+* hard: the tools parse the spilled stream and let the existing inline caps
+* bound what the model sees. A stream beyond even this budget still fails as
+* `SEARCH_RAW_OUTPUT_OVERFLOW`.
+*/
+const RAW_OUTPUT_SPILL_MAX_BYTES = 2e8;
 /**
 * Default cooperative tool-call timeout budget in milliseconds (the `timeoutMs`
 * config), attached to both tool definitions for
@@ -89,20 +99,41 @@
 	return new SearchError(`${toolName} search failed (exit ${exitCode})${stderr.length > 0 ? `: ${stderr}` : ""}`, "SEARCH_FAILED");
 }
 /**
-* Acquire the COMPLETE raw stdout of a finished run, enforcing
-* `rawOutputMaxBytes` on the in-memory transport. A truncated result means the
-* subprocess seam could not retain complete stdout within the requested
-* budget, so the tool fails clearly instead of parsing a silently-partial
-* stream.
+* Resolve the complete raw stdout of a finished run into a parse source,
+* enforcing `rawOutputMaxBytes` on the in-memory transport. Within the window
+* the retained text parses directly; an overflowed run instead streams its
+* COMPLETE spilled stdout ({@link RAW_OUTPUT_SPILL_MAX_BYTES}), so results stay
+* whole and the existing inline caps bound what the model sees. Only a stream
+* beyond even the spill budget - or a seam that kept nothing recoverable -
+* fails as `SEARCH_RAW_OUTPUT_OVERFLOW`.
 */
 function completeStdout(toolName, stdout, rawOutputMaxBytes) {
 	const narrow = "narrow pattern, path, or include and retry";
 	if (!stdout.lossy) {
 		const inlineBytes = Buffer.byteLength(stdout.text, "utf8");
 		if (inlineBytes > rawOutputMaxBytes) throw new SearchError(`${toolName} produced ${inlineBytes} bytes of raw output, over the ${rawOutputMaxBytes}-byte cap; ${narrow}`, "SEARCH_RAW_OUTPUT_OVERFLOW");
-		return stdout.text;
-	}
+		return { text: stdout.text };
+	}
+	if (stdout.spillPath !== void 0) return { spillPath: stdout.spillPath };
 	throw new SearchError(`${toolName} produced more raw output than the subprocess seam retained within the ${rawOutputMaxBytes}-byte cap; ${narrow}`, "SEARCH_RAW_OUTPUT_OVERFLOW");
+}
+/**
+* Yield one finished run's raw output line by line. Inside the in-memory window
+* this splits the retained text; an overflowed run streams its spill file
+* instead and deletes it when iteration ends. Callers own record parsing.
+*/
+async function* rawRunLines(stdoutResult) {
+	if (stdoutResult.spillPath === void 0) {
+		for (const line of stdoutResult.text.split("\n")) yield line;
+		return;
+	}
+	const lines = createInterface(createReadStream(stdoutResult.spillPath, { encoding: "utf8" }));
+	try {
+		for await (const line of lines) yield line;
+	} finally {
+		lines.close();
+		await unlink(stdoutResult.spillPath).catch(() => {});
+	}
 }
 let rgPathPromise;
 /**
@@ -136,9 +167,10 @@
 * The spawn is unconfined (a plain `ctx.subprocess` call), so `--no-config`
 * is prepended: a host `RIPGREP_CONFIG_PATH` (or `rg.conf` next to the
 * binary) can otherwise inject `--pre` and make ripgrep execute an arbitrary
-* preprocessor for every matched file. The collect dispositions are the
-* seam's diagnostic-tail shape (no spill files): the tools never read a raw
-* spill path, and truncated stdout fails as `SEARCH_RAW_OUTPUT_OVERFLOW`.
+* preprocessor for every matched file. The collect dispositions keep the
+* diagnostic stderr tail and add a stdout spill capture of the complete stream
+* within `RAW_OUTPUT_SPILL_MAX_BYTES`; only output beyond even the spill budget
+* fails as `SEARCH_RAW_OUTPUT_OVERFLOW`.
 *
 * Exit semantics are tool-owned: exit 0 is success with results, exit 1 is
 * success with zero results (`noMatches`), anything else throws a
@@ -174,7 +206,7 @@
 			cwd: workdir,
 			stdio: {
 				stdin: "ignore",
-				stdout: { maxBytes: rawOutputMaxBytes },
+				stdout: { maxBytes: rawOutputMaxBytes, spill: { maxBytes: RAW_OUTPUT_SPILL_MAX_BYTES } },
 				stderr: { maxBytes: stderrMaxBytes }
 			},
 			graceMs,
@@ -828,7 +860,7 @@
 				paths: []
 			};
 			const all = [];
-			for (const line of run.stdout.split("\n")) {
+			for await (const line of rawRunLines(run.stdout)) {
 				if (line.length === 0) continue;
 				const displayPath = toWorkdirRelative(line, run.workdir);
 				all.push(displayPath);
@@ -1139,7 +1171,10 @@
 			const run = await runRipgrep(ctx, exec, "grep", buildGrepCommand(parseGrepArgs(args)), caps.rawOutputMaxBytes, caps.graceMs, caps.stderrMaxBytes);
 			if (run.noMatches) return { matches: [] };
 			const all = [];
-			for (const raw of parseGrepMatches(run.stdout)) {
+			for await (const line of rawRunLines(run.stdout)) {
+				if (line.length === 0) continue;
+				const raw = parseRecord(line);
+				if (raw === void 0) continue;
 				const match = {
 					path: toWorkdirRelative(raw.path, run.workdir),
 					lineNumber: raw.lineNumber,

---

Environment

DSH desktop install; @deepseek-ai/* 0.1.1-rc.2 (patches applied to the compiled lib/index.js); Windows. Both patches verified red/green with independent harnesses that mirror the seam/compaction semantics against the real packaged rg binary.

[Saidoua]

A complementary approach to the same failure, shared under Show Your Plugins: an in-process backend for tool-fs-search on ripgrep's library crates, so there is no --json transport to overflow — grep "const" over packages/ (21 MB of --json) returns 75 108 matches instead of SEARCH_RAW_OUTPUT_OVERFLOW. Your spill-backed parse is still the right in-tree fix for the spawn path; this removes the spawn. Measurements and parity suite: #5685