perf(util): add an incremental partial-JSON stream parser

[sroussey]

Closes #507

The literal ask was already done; the stated goal was not

A partial-JSON parser already exists — packages/util/src/json-schema/parsePartialJson.ts, tested and in use. So "add a json streaming parser" reads as satisfied.

But the issue's stated goal is "so we don't have to accumulate structured output", and that never happened. parsePartialJson is a repair-and-reparse of a truncated buffer, not a streaming parser: every call runs JSON.parse on the whole buffer, then re-scans the whole buffer character-by-character to repair it, then JSON.parses again. There is no cursor, no container stack, no push()/finish().

Accumulation was therefore never eliminated — it was relocated into every provider run-fn. All eleven *_StructuredGeneration.ts files held a growing accumulatedJson string and called parsePartialJson(accumulatedJson) once per delta. That is O(n²), and run-fns execute inside workers, so it fully blocks a worker thread.

Measured on the built bundle (packages/util/dist), accumulate-and-reparse vs. the new parser:

payload	chunk size	chunks	before	after	speedup
24 KB	8 chars	3129	3999 ms	12.3 ms	326x
98 KB	40 chars	2502	9512 ms	9.8 ms	971x

4.1x the bytes costs ~2.4x the time before (super-linear); after, it is flat.

What this adds

createPartialJsonStream(options?) in packages/util/src/json-schema/PartialJsonStream.ts — a resumable tokenizer holding a container stack of live JS objects/arrays, the pending key, a scalar buffer bounded by the current token (not the document), and string-escape / \uXXXX state. push(chunk) is O(chunk.length) and never re-scans consumed input.

• push(chunk) → the live root once a top-level object has opened, else undefined
• finish() → closes open containers/strings; idempotent
• snapshot() → detached deep copy
• complete → whether the root closed on its own
• skipPreamble option → discards text before the first {/[, for providers that emit prose or a <think> block first

parsePartialJson is exported and unchanged — it remains correct and in use for one-shot repair call sites.

⚠️ Behavioral tradeoff reviewers must consent to: live-root aliasing

push() returns the parser's live root object, which subsequent pushes keep mutating. This is the one intentional behavior change, and it is what makes the parser O(n) — deep-cloning on every delta would reinstate the O(n²) this PR exists to remove.

Why it is safe on the paths we actually use:

• StreamEventAccumulator.observe and StreamProcessor use replace semantics for non-array object-deltas, so repeatedly storing the same growing object converges to the correct final value.
• Events are structured-cloned crossing the worker boundary, so out-of-process consumers get independent copies anyway.

Where it bites: an in-process consumer that retains an earlier delta will see that object change under it. The aliasing is documented in the interface JSDoc, snapshot() is the escape hatch for anyone needing a frozen copy, and there is a test asserting each behavior explicitly.

Guard preserved: push() returns undefined for a top-level array root (matching parsePartialJson), so no run-fn can emit an array as an objectDelta — that would hit the accumulators' append-by-id branch and re-append the whole growing array each delta.

Why the parser stays in @workglow/util, not @workglow/ai

The issue says "to the ai package", but that placement would be wrong here. util is the foundation package, and the consumers are provider run-fns that already import worker-safe helpers from @workglow/util/worker — @workglow/ai sits above them in the dependency graph. Moving it would break that import path and buy nothing. It is exported from both schema-entry.ts and worker-entry.ts, exactly like parsePartialJson.

Migrated call sites

All 11 *_StructuredGeneration.ts run-fns: anthropic, openai, google-gemini, ollama, xai, deepseek, openrouter, chrome-ai, node-llama-cpp, huggingface-transformers, tf-mediapipe. Each collapses accumulatedJson + per-delta parsePartialJson + the trailing try { JSON.parse } catch { parsePartialJson } into one stream instance ending in emit({ type: "finish", data: { object: json.finish() } }).

Notes:

• HFT / TFMP additionally did parsePartialJson(fullText.slice(jsonStart)) per delta; both now use skipPreamble and drop the jsonStart bookkeeping and the per-delta .slice(). Their existing behavior of emitting text-delta while no JSON has appeared yet is preserved (detected via push() returning undefined). HFT's now-dead extractJsonFromText/stripThinkingAndSpecialTokens helpers are removed; TFMP's extractJsonFromText is exported and tested, so it stays.
• chrome-ai emits progressive full-text snapshots rather than deltas; it keeps its snapshot diff and feeds only the newly-appended tail, restarting the parser on the rare replacement case.
• deepseek's assertNotTruncatedByReasoning only distinguishes "no content" from "some content", so it now gets one retained delta instead of a full second copy of the document. The helper's public signature is untouched.
• The CLAUDE.md rule that json-mode run-fns MUST populate finish.data.object stays in force — json.finish() satisfies it, and it remains the validation anchor for StructuredGenerationTask's retry loop. The stale rationale ("avoids requiring a JSON streaming parser in the consumer layer") is replaced with the real reason plus the aliasing note.

Two incidental correctness wins over parsePartialJson, both asserted as tests: a mid-key prefix keeps the object built so far instead of discarding the whole object, and trailing whitespace inside an in-flight string value is preserved instead of being trimmed away. __proto__ is written as an own property via defineProperty, matching JSON.parse and avoiding prototype pollution from untrusted model output.

Deliberately NOT in scope

The *_ToolCalling.ts call sites (anthropic, ollama, llamacpp-server, and the provider-utils shared helper) still use parsePartialJson. Open PR #641 edits those files, so migrating them here would conflict. Follow-up:

• providers/anthropic/src/ai/common/Anthropic_ToolCalling.ts
• providers/ollama/src/ai/common/Ollama_ToolCalling.ts
• providers/llamacpp-server/src/ai/common/LlamaCppServer_ToolCalling.ts
• packages/ai/src/provider-utils/OpenAIShapedChat.ts

Verification

All run on this branch; every command's real result:

command	result
bun run build	84/84 tasks successful
bun run build:types	41/41 tasks successful
bun scripts/test.ts util vitest	47 files passed, 711 passed / 10 skipped
bun scripts/test.ts ai vitest	38 files passed, 247 passed
bun scripts/test.ts provider-api unit vitest	38 files passed, 410 passed
bun scripts/test.ts provider unit vitest	16 passed / 1 skipped, 226 passed / 5 skipped
npx eslint on changed files	clean

packages/test/src/test/util/PartialJsonStream.test.ts adds 39 tests:

1. Chunk-boundary exhaustive — 11 documents (nested objects, arrays, escapes, unicode, numbers, null/true/false, empty containers, pretty-printed whitespace) split at every 1-character boundary, plus fed one character at a time; finish() deep-equals JSON.parse(doc) in every case.
2. Equivalence with parsePartialJson at every prefix — where it produces a result the stream matches it, and the stream never loses a result it would have produced. Prefixes ending in whitespace are excluded because parsePartialJson trims its input; both intentional divergences are asserted explicitly as their own tests.
3. Split escapes — "a\ + "b", \u00 + 41, \\ at a chunk edge, a surrogate pair, and a \uXXXX document split at every internal boundary.
4. Split scalars — 1. + 5e + -3, tr + ue, nul + l, and an exponent terminated by end of stream.
5. Truncated streams — finish() closes open containers and drops trailing incomplete tokens, matching the current repair behavior ({"a": tru → {}, {"n": 30 → {n:30}); idempotency asserted.
6. Preamble skipping — Here you go: {"a":1}, a <think>…</think> prefix, a markdown fence, trailing commentary, and recovery when a <think> block itself contains braces.
7. Perf regression guard — 100 KB in 40-char chunks must finish under 250 ms (measured ~10 ms; the old path takes multiple seconds), so only a genuine algorithmic regression trips it.

packages/test/src/test/ai-provider-api/Ollama_StructuredGenerationStream.test.ts — which specifically asserts the no-closing-brace recovery path — passes unchanged, as does parsePartialJson.test.ts.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PwyJuFrJnibKvrrk8Fa4Fn

---

Generated by Claude Code

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	64.35%	31731 / 49306
🔵	Statements	64.15%	32830 / 51170
🔵	Functions	65.43%	5975 / 9131
🔵	Branches	53.33%	16345 / 30648

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
providers/huggingface-transformers/src/ai/common/HFT_StructuredGeneration.ts	2.56%	0%	0%	2.63%	24-28, 37-105

Generated in workflow #2891 for commit fb7c7b3 by the Vitest Coverage Report Action