Bug: session search crashes with RangeError: Invalid string length on large sessions (root cause + fix)

[flyingcoding]

session-query SQLite: workspace search crashes on large sessions (RangeError: Invalid string length)

Summary

session_search / session_event_search return Error: session query operation failed once any session in a workspace grows large enough. The underlying error is a RangeError: Invalid string length thrown during search-index reconciliation, and it takes down search for the entire workspace, not just the big session. Session listing and exact event reads keep working, which hides the failure for a while.

Root cause

observeSession() in packages/session-query/session-query-sqlite/src/index.ts computes the change fingerprint by serializing the whole event log into one string:

fingerprint: createHash('sha256')
  .update(JSON.stringify({ header: detachedHeader, events: detachedEvents }))
  .digest('base64url'),

V8 caps a single string at ~512 MB (2^29-1 characters). With a session holding millions of events (observed live with ~5M events; a persisted log of ~150 MB raw / 223k events is already enough), the stringify output crosses the cap and JSON.stringify throws:

RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at observeSession (.../dsh-session-query-sqlite/lib/index.js:1000:49)
    at observeLive (...:992)
    at Proxy._observeStable (...:740)
    at async Proxy._reconcile (...:651)

The reconciliation failure aborts the query, so every search in that workspace fails. The tool layer additionally masks the real error behind the generic session query operation failed message (SESSION_QUERY_INVALID_CONFIG / SESSION_QUERY_SOURCE_CONFLICT mapping in tool-session-query/service-boundary.ts).

Reproduction

1. Run a session long enough that its event log stringifies beyond ~512 MB (long-running agent loop with heavy tool traffic works well).
2. Call session_search in that workspace.
3. Observe Error: session query operation failed for any query, while session listing and exact reads still work.

Proposed fix

Hash the fingerprint incrementally — header once, then each event separately — so memory stays bounded:

function observeSession(header: SessionHeader, events: readonly SessionEvent[]): ObservedSession {
  const detachedHeader = structuredClone(header)
  const detachedEvents = events.map(event => structuredClone(event))
  // Incremental fingerprint: avoid one giant JSON.stringify of the whole log,
  // which exceeds V8's max string length (~512 MB) on multi-million-event sessions.
  const fingerprintHash = createHash('sha256').update(JSON.stringify(detachedHeader))
  for (const event of detachedEvents) {
    fingerprintHash.update('\n')
    fingerprintHash.update(JSON.stringify(event))
  }
  return {
    header: detachedHeader,
    documents: buildSessionEventSearchDocuments(detachedHeader.id, detachedEvents),
    fingerprint: fingerprintHash.digest('base64url'),
  }
}

Side effect: fingerprints change once for every session (one-time index rebuild), which is expected whenever the hash construction changes.

Verification

• Replayed the incremental fingerprint over the real 223k-event persisted log that crashed the old code — completes without error.
• Patched the installed dsh-session-query-sqlite/lib/index.js and restarted dsh web: both session_search and session_event_search work again in that workspace, including over the large session.

Observed on @deepseek-ai/dsh-session-query-sqlite 0.1.0-rc.6 (published dsh family); the same pattern is present at master in this repository.

[zoahdev]

Endorsing the incremental-fingerprint fix ? the V8 cap mechanism checks out, and the error-masking in service-boundary.ts is worth fixing in the same pass so a reconciliation failure never surfaces as a generic session query operation failed again.

Two additions to the fix:

1. structuredClone of the whole event list is still O(n) memory (and events.map(clone) doubles it transiently). For multi-million-event sessions, streaming hashing helps the string cap but the clone array can still pressure the heap. Consider cloning lazily per event inside the loop (clone ? hash ? drop), or batching.
2. Diagnosability: a preemptive session-size signal would catch this before the 512 MB cliff. Happy to add a session-size warning to a doctor-style check (or the ecosystem's session tools) that reports the largest session's event count/raw log size so operators get warned early.

The reproduced 223k-event threshold (~150 MB raw) is a great regression fixture ? worth adding to session-query-sqlite tests.

[zoahdev]

The preemptive signal is now shipped: dsh-plugin-doctor v1.10.0 adds a large-files check to --profile ? warns when any profile file exceeds 100 MB (skipping node_modules/.pnpm), naming the relative path and size, with a pointer to this discussion.

npx dsh-plugin-doctor --profile ~/.dsh/profiles/web --json
# ? large-files: warn | sessions/huge.jsonl (150.0 MB) ?

25/25 tests (3 new fixtures: under threshold PASS / over threshold WARN with path / dependency-tree skip). Release: https://github.com/zoahdev/dsh-plugin-doctor/releases/tag/v1.10.0 ? the upstream incremental-hash fix remains the real cure; this catches the cliff early.