Skip to content

fix: bound provider telemetry memory - #3168

Merged
numnx merged 2 commits into
devfrom
fix/node-heap-exhaustion
Jul 17, 2026
Merged

fix: bound provider telemetry memory#3168
numnx merged 2 commits into
devfrom
fix/node-heap-exhaustion

Conversation

@numnx

@numnx numnx commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • Bound Codex rollout parsing to 2 MiB per JSONL record, capped message/tool fields, and the newest 256 conversation groups.
  • Serialize full-history Jules telemetry process-wide and deduplicate concurrent reads for the same session.
  • Tokenize Jules transcripts and patches in slices of at most 64 KiB, build bounded patch messages without interpolating the full patch, and release raw activities before SQLite reconciliation.
  • Add regression tests for oversized Codex records, long rollout histories, Jules synchronization concurrency, tokenizer input bounds, and large patch persistence.
  • Document the runtime memory bounds in canonical and published architecture guidance.

Root cause

The V8 abort was application heap exhaustion, not a kernel OOM kill. Two independent amplification paths were present:

  • PR fix: bind provider resumes to persisted sessions #3166 correctly bound Codex telemetry to the exact native rollout, but exposed an older parser weakness: an incomplete JSONL record and parsed tool payloads had no retained-size bound.
  • Live Jules usage sync launched fire-and-forget full-conversation reads for many sessions concurrently, then passed multi-megabyte patches to js-tiktoken and constructed a full interpolated patch string before truncation.

The Codex parser weakness was related to the latest PR commits; the Jules synchronization path predated them and was uncovered after the first fix was exercised against the live recovered workload.

Impact

Large generated assets, command output, and wide hosted-session synchronization no longer multiply into an unbounded server heap spike. Pathological Codex records are omitted from the dashboard transcript, but parsing resumes at the next record and usage/native session identity remain available. Jules usage and message persistence retain their existing contract, with full-history refreshes processed one at a time.

Validation

  • pnpm run ci passed
  • Full Vitest suite: 9,804 passed, 13 skipped
  • Backend coverage thresholds passed
  • Focused Codex/Jules memory suites passed
  • 514 MiB synthetic oversized-record stream completed under a 128 MiB V8 heap limit; parser recovered the following record and settled at 6 MiB heap
  • The compiled runtime replayed recovered orchestration for more than seven minutes under a 512 MiB V8 heap limit; /ready remained healthy, JS heap stayed around 170–203 MiB, and RSS stabilized around 0.70–0.75 GiB

Risk and rollback

Wide Jules batches can delay an individual live transcript refresh while earlier sessions finish, but they no longer run full-history fetches concurrently. Codex usage accounting and session selection remain unchanged. Roll back by reverting this PR; no database migration or persisted-data rewrite is involved.

@numnx numnx changed the title fix: bound Codex rollout parser memory fix: bound provider telemetry memory Jul 17, 2026
@numnx
numnx marked this pull request as ready for review July 17, 2026 16:14
@numnx
numnx merged commit 430ac97 into dev Jul 17, 2026
40 checks passed
@numnx
numnx deleted the fix/node-heap-exhaustion branch July 17, 2026 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant