experimental-session-search: every call fails with SESSION_QUERY_PERSISTENCE_FAILED (0.2.1-rc.2, self-hosted profile)

[Ersith]

Environment

• DSH Desktop 0.2.1-rc.2 (win-x64, Windows 11), Electron-bundled node v24.21.0
• Self-hosted profile desktop (DSH_HOME outside the default location)
• Session store: ~220 sessions / 407 MB under sessions/
• Third-party plugin set installed — see the last section

Symptom

Enabling 会话搜索 (@deepseek-ai/dsh-experimental-session-search) in the plugin page does register its tools (all 5 appear), but every call fails:

Error: session history storage is unavailable

From the packaged error table this is SESSION_QUERY_PERSISTENCE_FAILED:

throw new SessionQueryError(`session-search persistence observation failed: ${errorMessage(error)}`, "SESSION_QUERY_PERSISTENCE_FAILED", { cause: error });
// and
throw new SessionQueryError("session-search persistence observation did not stabilize after one retry", "SESSION_QUERY_PERSISTENCE_FAILED");

Reproduction (deterministic, 3 ways)

1. Enable the bundle → call session_search with any query → fails.
2. Enable the bundle → fully restart the app → call session_search → fails.
3. Same as (2), calling it immediately after a fresh start, before any turn writes anything → fails.

So it is not a "first call while a session is being written" race, at least not only that.

Composition verified via cordis_inspect_query (host Config provider)

entry	status
include:session-query-sqlite (base)	live
include:optional-session-query-sqlite (bundle)	live
include:optional-tool-session-query	live
include:session-persistence-jsonl	live

There is no .sqlite file in the profile — expected, since the bundle's patch sets path: ':memory:', openAt: first-search, so the index is in-memory and lazily opened. (Mentioning it because it was my first wrong guess.)

Hypotheses we ruled out

1. "Two rows mount the same module ⇒ conflict." No — the bundle's cordis.patch.yml uses isolate: { sessionQuery: true } on the group, so this is by design.
2. "A malformed session archive makes the cold-log reader throw." We found 5 bad dirs (3 × 0-byte session.v4.jsonl.zstd, 1 dir with no .zstd at all, 1 more 0-byte), quarantined them (moved aside, not deleted); afterwards 220/220 archives read cleanly — we re-read the whole corpus (407 MB) with an RFC 8878 frame-aware zstd reader: 0 bad frames, 0 truncated. Still fails.
3. "Reconcile never stabilizes because a large corpus takes long and a live session is appended during it." We measured a full cold read of the corpus at 14.4 s, which makes this plausible in principle — but (3) above shows it also fails on the very first call after a restart.

Questions

1. Is SESSION_QUERY_PERSISTENCE_FAILED the expected outcome when the provider inside an isolate: { sessionQuery: true } group cannot see sessionPersistence? The provider acquires it via ctx.inject(["sessionPersistence"], …). Is there a known composition where that optional injection never fires inside an isolated group?
2. The model-facing tool swallows cause — we only get the generic message, so we cannot distinguish readColdSessionLog failing from "did not stabilize after one retry". Is there a documented debug flag / log line to surface the underlying error?
3. Is there an intended config for the bundle on a self-hosted profile that also mounts session-persistence-jsonl, that we are missing?

Third-party plugins in the same profile (possible interference?)

We run a local plugin set —akasha-book (@local/akasha-gate, akasha-hooks, akasha-session, akasha-sleep, mcp-akasha, source: https://github.com/Ersith/akasha-book). It is an external-memory project; its session layer reads the same .zstd session archives and maintains its own index, but it does not register sessionQuery and does not write to the session store.

Two things we want your read on:

• Could a plugin that reads the same sessions/ directory (and holds open read handles / stats files while the provider is taking its two snapshots) make persistence.list() snapshots differ, or otherwise interfere with readColdSessionLog?
• We are happy to run a clean reproduction with the whole plugin set disabled, if that is useful.

Versions

• @deepseek-ai/dsh-desktop 0.2.1-rc.2
• @deepseek-ai/dsh-session-query-sqlite, @deepseek-ai/dsh-tool-session-query, @deepseek-ai/dsh-session-persistence-jsonl — shipped with the same build
• node v24.21.0 (runtime), v24.19.0 (system toolchain)

Thanks!

[IlyaKhalafi]

There is already a host-side diagnostic for this. I checked the published 0.2.1-rc.2 packages. The tool boundary logs tool-session-query: <operation> failed: ... at warning level, including the original stack and cause chain, before replacing the model-facing error. Look for that warning from one failing call. It should distinguish the underlying persistence exception from did not stabilize after one retry.

A missing sessionPersistence binding alone does not take either of those failure paths. The observation loop skips persisted reads when that service is absent and proceeds with live sessions. It can still fail if live membership keeps changing. So the generic message does not establish an isolation/injection problem.

For the instability variant, the comparisons include persisted snapshots, persistence-binding identity and live session IDs. A newly attached session can matter even before a turn writes anything. For the exception variant, valid Zstandard framing does not rule out a failure in persistence.list() or DSH's cold-log read.

A clean-plugin run is useful as a comparison, but capture the warning first. That identifies which branch to investigate without guessing at configuration or moving more archives. The code links show the corresponding source; this does not reproduce your Windows failure or identify its cause.

[Ersith]

Follow-up: reproduced the underlying exception with the installed official modules

We have now captured a concrete cause without killing/restarting the desktop host or modifying the production profile.

The probe uses the bundled Node v24.21.0 and the installed 0.2.1-rc.2 Context, SessionStore, JSONL persistence and SQLite query modules. No third-party plugins or live sessions are present. Every session handle is opened with access="read", and SQLite uses :memory:. We also compared the extracted modules against the running desktop's app.asar: all 82 files across nine relevant packages are byte-identical.

Searching the unchanged corpus reproducibly fails in under 0.5 seconds:

SessionQueryError [SESSION_QUERY_PERSISTENCE_FAILED]
  session-search persistence observation failed: ...
Caused by: SessionFormatUnsupportedError
  contains event type "request/plan" (seq 37) unknown to this harness
  and not marked ignorable

A scan using the official readColdSessionLog — rather than our Zstandard frame reader — found five rejected sessions among 237 enumerable sessions:

• One V4 archive: unknown required request/plan event at seq 37. Its payload is a request-assembly plan containing an out-of-repo plugin's resources; we have not identified the emitting version.
• Three V0 archives: released-row sequence gaps.
• One V0 archive: cannot safely transform unclassified message source.

This corrects our earlier statement that malformed/incompatible archives had been ruled out. Valid compression frames do not establish that DSH's logical reader accepts the records.

Controls:

1. Filtering only those five IDs out of persistence.list() inside the isolated probe allows full-corpus search to succeed. The production files remain untouched. One retry occurred after a new session appeared; the next observation stabilized and succeeded.
2. A copied healthy archive searches successfully. Adding a byte-identical copy of the request/plan archive to that small static corpus reproduces the same error and cause.
3. Omitting persistence entirely returns an empty search result, rather than this error.

One point we would like to confirm: dsh-session-persistence-jsonl documents the per-event ignorable: true envelope marker as the compatibility mechanism for event types outside the build's vocabulary (and its plugin-record append path sets it automatically). The offending request/plan record does not carry it. Is marking such a record ignorable: true — preserving the event, its data and its seq — the supported lossless recovery path for external/plugin events, or is a dedicated migration/reader change expected?

The remaining questions are the supported recovery/compatibility path for these archives (especially the three V0 sequence-gap records and the one unclassifiable V0 source) and whether global search should expose the specific rejected session or support explicitly reported partial results instead of failing the entire query.

We have not rewritten or quarantined production archives, and have not restarted the desktop host.

[Ersith]

Update: local repair performed and search restored — with two implementation findings

We applied a local, reversible repair (full original archives backed up) and the desktop session search now works again. Summary, in case it helps the upstream discussion:

• 4 of the 5 rejected archives were repaired and restored; each passes the official readColdSessionLog (full-corpus check: 238/238 sessions readable).
• Repairs were additive/referential, not destructive:
  • the V4 request/plan record was kept and marked ignorable: true on the event envelope (data and seq unchanged);
  • 11 user/message sources of kind instruction-hint (a plugin-injected hint) were marked as plugin so the v2→v3 source rewrite can map them;
  • two V0 logs with a duplicated seq row were renumbered from the duplicate onward, remapping sourceEventSeqs / messageSeqs / surfaceOp references.
• One V0 archive remains quarantined: its seq rewinds by 8224 in a later row, and that segment’s references overlap the earlier segment’s seq range, so a purely numeric remap is ambiguous (tool/result ... sources targets consumed assistant/chunk ...). We did not rewrite it.

Two findings that may be worth documenting:

1. persistence.list() does not enumerate an archive that was rewritten as a single Zstandard frame; preserving the original multi-frame layout made the repaired V4 archive visible again. (A naive re-compression to one frame silently drops the session from the list.)
2. The indexed document scope is narrower than “all event text”: content carried only in agent/inbox/spliced inserted messages, or in request/plan/request/context payloads, is not surfaced by search. Two of the repaired historical sessions contain their human-readable user messages only in those slots, so they remain listable but not searchable by content.

Question: would you expect local repairs of this kind (marking external records ignorable, renumbering duplicated seqs with reference remapping) to be supported by a documented migration/repair path, rather than ad-hoc scripts? And should the tool error expose the first rejected session id (instead of only the sanitized message) to make this diagnosable from the tool surface?

[Ersith]

Correction to the previous update: three of the five archives are repaired and restored, not four.

The large V0 archive I reported as repaired (109,342 frames) was re-quarantined after a full-corpus check. Root cause of the correction: after fixing the 11 instruction-hint sources, the V0→V4 migration read (prepareStoredMigration → streaming Zstandard decoder) reached frame 106,174 and reported:

ZSTD_error_checksum_wrong: Restored data doesn't match checksum
frame at byte 36,518,857

That frame is part of the original artifact — the pre-existing sequence of the V0 reader stopped earlier at the source-kind error, so this latent checksum corruption had never been reached. The source-kind repair exposed it.

Two implementation notes from this:

1. readColdSessionLog’s ordinary read path did not report the checksum failure (it returned the session successfully); only the migration streaming path validates the frame checksums. Verification of a rewritten V0/V1 archive therefore has to go through the migration/reconcile path, not only the cold-read path.
2. The archive is now re-quarantined with its original bytes preserved. Repairing it would require decoding the affected frame, rewriting it with a correct checksum (while keeping the multi-frame layout), and re-running the full read.

Current state: 3 of 5 repaired and restored (the V4 ignorable marking plus two V0 seq-renumbering repairs), 2 quarantined (this checksum-corrupt one and the seq-rewind archive with ambiguous references). Full-corpus session search is operational again. I’ll follow up if the checksum-corrupt frame can be repaired safely.

[Ersith]

Follow-up after the mitigations you may not have tried yet — the failure mode changed, and the new data narrows it to the observation-stability requirement.

Status change: from "every call fails" to "intermittent"

After the initial report we changed two things in our profile:

1. Persistent derived index instead of path: ':memory:' — the bundle was cold-building the whole corpus on every start (~70–80 s; our store is ~220 sessions / 400 MB), and the first search was being cut off by the tool timeout. We now point path at a real file on disk and keep a 364 MB sqlite next to the profile.
2. searchTimeoutMs: 180000 on the tool row (the shipped default 30 s was too short for the first reconciliation).

With that, "every call fails" is gone, but session search is now intermittent: 13 failed calls in ~10 hours on 0.2.1-rc.2 (win-x64), while the surrounding calls succeed.

Failure distribution

Of the 13 failures, 12 surface as session history storage is unavailable (SESSION_QUERY_PERSISTENCE_FAILED) and 1 as session search index is unavailable (SESSION_QUERY_INDEX_FAILED). Timestamps (UTC, clustered):

2026-10-17T11:07:58Z  PERSISTENCE_FAILED
2026-10-17T11:07:58Z  PERSISTENCE_FAILED
2026-10-17T13:05:43Z  PERSISTENCE_FAILED
2026-10-17T13:07:46Z  PERSISTENCE_FAILED
2026-10-17T14:45:46Z  PERSISTENCE_FAILED
2026-10-17T14:51:21Z  PERSISTENCE_FAILED
2026-10-17T20:06:04Z  PERSISTENCE_FAILED
2026-10-17T20:06:51Z  PERSISTENCE_FAILED
2026-10-17T21:07:03Z  PERSISTENCE_FAILED
2026-10-17T21:46:21Z  INDEX_FAILED
2026-10-17T21:48:04Z  PERSISTENCE_FAILED
2026-10-17T21:48:56Z  PERSISTENCE_FAILED
2026-10-17T21:49:26Z  PERSISTENCE_FAILED

The 21:48–21:49 cluster is the one we can correlate: host event logs show another session doing turn-end / tool work in the same seconds, i.e. the session store was being written while the observation ran. (Indirect, but consistent with #9743's "live-session churn prevents the observation from stabilising".)

Code reading (packaged 0.2.1-rc.2)

• Every searchSessions() / searchEvents() call goes _ensureReady() → _reconcile() → query, so this is not a first-search-only problem.
• _reconcile() → _observeStable(): it lists persistence twice and requires samePersistenceSnapshots(before, after). Any session whose revision changes between the two listings aborts the attempt.
• STABLE_OBSERVATION_ATTEMPTS = 2 is hard-coded — one retry — then throw new SessionQueryError("session-search persistence observation did not stabilize after one retry", "SESSION_QUERY_PERSISTENCE_FAILED").
• So on a profile with a continuously active session, a search that happens to overlap a turn-end has a real chance of exhausting both attempts.

Mitigations measured (for the record)

• Persistent index removed the repeated cold build; first search after restart no longer hits the timeout.
• openAt: startup was considered but not adopted: on the 364 MB index it costs ~1.4 ms to open (measured), so it is cheap, but it does not touch _observeStable() — it only moves an _open() failure from first-search into host startup, where [Service.init] awaits it without a catch. Net negative.
• Client-side retry (2 attempts) is what we do today; it hides the symptom but does not fix it.

Suggestions (small, code-level)

1. Make STABLE_OBSERVATION_ATTEMPTS configurable, or raise the default (e.g. 4) with a small backoff between attempts.
2. Narrow the stability comparison: require a stable snapshot only for sessions that are actually being (re)indexed this round — or accept "changed sessions are simply re-read next round" instead of failing the whole call. canReuseIndexed already skips unchanged revisions, so a quiescent global snapshot may be stricter than needed.
3. Keep INDEX_FAILED and PERSISTENCE_FAILED distinguishable at the tool boundary (or append a short cause), since "storage is unavailable" currently points users at the wrong layer.

One question on the diagnostic you mentioned

You noted the tool boundary logs tool-session-query: <operation> failed: ... at warning level with the original stack and cause chain. We could not find that log on disk — %APPDATA%\@deepseek-ai\dsh-desktop\logs\ only contains crash-*.log, and nothing under the profile home contains those lines. Could you point at the default sink, or the flag/env that enables it? With the real cause we can tell readColdSessionLog failures apart from snapshot instability, which would make the next report much sharper.

Related: #9743 covers the "churn prevents stabilisation" half of this in detail; this report adds the 0.2.1-rc.2 after-mitigation numbers and the failure clustering.

[IlyaKhalafi]

My earlier answer needs a qualification. The warning call exists, but that does not establish a persistent log sink. The desktop crash reports concern fatal failures. The optional console exporter writes to stdout, which Desktop forwards to its own stdout, not that crash directory.

There is also a verbosity trap. Warnings have level 2, while the fallback threshold is INFO, level 1. In your isolated probe, register ctx.logger.exporter on the same root context with levels: { default: 2 }, collect warning records whose first string argument starts with tool-session-query: , and dispose the exporter afterward. Exercise the tool boundary, since a direct provider call will not produce that boundary warning. Keep the full diagnostics local and redact before sharing.

Your churn correlation is consistent with the two-attempt limit, but the captured cause is still needed to identify those intermittent failures. A longer timeout does not increase that limit. Extra bounded retries could help transient churn; narrowing snapshot checks needs consistency tests around concurrent updates, deletion and persistence-binding changes.

[Ersith]

Captured the cause. Instrumenting the tool boundary with a logger exporter (ctx.logger.exporter, levels: { default: 2 }, filtering records whose first string argument starts with tool-session-query-) worked exactly as you described — that's the only place this warning is visible.

Reproduced within ~3 seconds once I added churn. The record:

tool-session-query: session search failed: SessionQueryError: session-search persistence
observation did not stabilize after one retry
  at Proxy._observeStable   (…/dsh-session-query-sqlite/lib/index.js:755:9)
  at async Proxy._reconcile (…/index.js:653:23)
  at async …/index.js:543:31
  at async Proxy._serialized (…)

Reading _observeStable around that line, the unstable condition looks narrower than "the persistence snapshot changed": the final check compares the session-id set between the two observations (if (!sameSessionIds(initiallyLive, live)) continue;). So it trips when a session is created or destroyed while reconcile is in flight — content changes alone don't do it. persistence.list() itself looked fine in both samples.

Reproduction here:

• index path set to :memory: (dropping our persistent index only lowers the bar; same failure, faster — it is not the cause)
• 5 concurrent sessions, each doing ~20 s of work (each one is a session creation, plus file writes)
• 1 of 2 concurrent searches failed on the first attempt; ~10 earlier searches (3 batches) with a single session all passed

So this is consistent with your churn reading, but the trigger is session-set churn, not persistence-snapshot churn.

Fix suggestion, if useful: treat additive changes as stable — a newly created session can just be indexed incrementally, instead of abandoning the whole observation and retrying (which currently buys exactly one more try). Only removals, or a persistence-binding change, would need the retry. Failing that, making STABLE_OBSERVATION_ATTEMPTS / the observation interval configurable would at least let us trade latency for stability.

Environment: Windows desktop, runtime 0.2.1-rc.2. Raw records are retained locally and I can share them (paths redacted) if that helps.