[Bug report] One persisted session with an unsupported event (demo/hello) can abort global reconciliation and make session search report storage unavailable

[yangfei222666-9]

Summary

During this incident, all session-search calls we tested returned:

session history storage is unavailable

The loopback HTTP endpoint and other functions we checked remained responsive. The failing corpus contained two historical persisted-session artifacts; each was rejected by the installed runtime as unsupported on demo/hello at sequence 3.

This report documents the connection between the same unknown-event incompatibility reported in #3802 / #3416 and the global SQLite session-query reconciliation path. It does not claim first discovery of the underlying event incompatibility or of the general one-bad-artifact blast-radius pattern.

Environment and observed scope

• macOS desktop-managed DSH
• @deepseek-ai/dsh-tool-session-query: 0.1.0-rc.7
• session/query/persistence core packages: 0.1.1-rc.2
• base/web-app: 0.1.1-rc.1
• JSONL persistence with the SQLite session-query backend

Before containment, PRAGMA quick_check returned ok, but the SQLite index held only 2 persisted-session rows while 33 persisted artifacts were present. The installed provider could inspect 31 of those 33 artifacts; exactly two threw SessionFormatUnsupportedError on demo/hello at sequence 3.

These are incident observations from the installed mixed prerelease runtime. The source trace below is pinned to upstream HEAD at filing time; we are not claiming that the installed package bytes are identical to that commit.

Root-cause chain

At upstream HEAD cd5ef8148158c3a752a658978873241fdf8e2bbc:

1. The persistence layer classifies an intact log that this runtime cannot interpret as SessionFormatUnsupportedError, distinct from corruption: coordinator.ts#L48-L64.
2. After stored-event normalization, assertEventsSupported() rejects an event type unknown to the build: coordinator.ts#L1132-L1152.
3. When a persisted entry requires inspection, SQLite observation calls inspect() without per-entry isolation. One throw aborts that observation pass and is wrapped as SESSION_QUERY_PERSISTENCE_FAILED: index.ts#L483-L531.
4. _reconcile() awaits the observation before BEGIN IMMEDIATE, so this failure precedes the index writes in that reconciliation transaction: index.ts#L395-L442.
5. The tool boundary maps that code to the generic message session history storage is unavailable: service-boundary.ts#L70-L73.

The live incident contained two incompatible artifacts and both were contained together. The loop has no per-entry catch, so the code path shows that one inspected entry throwing is sufficient to abort the pass; we did not separately run a one-bad-artifact live test.

Impact

The failure prevents a search response from being produced from otherwise inspectable entries, and the generic message looks like a storage outage rather than a per-session format incompatibility.

Minimal local containment and result

We froze the SHA-256 values of the two failing artifacts, moved exactly those two session directories out of the active scan into a reversible quarantine, and verified unchanged hashes after the move. We did not delete either artifact or rewrite its contents.

The active artifact count changed only from 33 to 31. After containment, a live session_search returned successfully in our environment.

This is containment, not an upstream source fix.

Suggested behavior

Please keep interpretation fail-closed for the affected session, but do not fail the entire corpus. During reconciliation, catch only SessionFormatUnsupportedError at the per-entry boundary; mark that entry incompatible, atomically exclude or tombstone any stale indexed rows for it, leave the artifact untouched, emit an actionable warning with its session/location, and continue with otherwise inspectable entries. The caller should receive an explicit partial/degraded indication rather than silently incomplete results. Other persistence errors should continue to fail normally, and unknown events should not be silently ignored.

Separately, a formal registration or ignorable-event surface for out-of-repo event types (as discussed in #4204) could prevent this trigger for explicitly supported plugin events, but it would not replace per-entry isolation for other unsupported artifacts.

Claim boundaries

• We are not claiming the two artifacts are corrupt; the evidence supports format unsupported/incompatible.
• We have not attributed the writer or originating test for demo/hello.
• The live incident had two failing artifacts; one-entry sufficiency is based on the pinned control flow, not a separate one-bad-artifact live test.
• The pinned HEAD trace is static source evidence, not proof that our installed mixed-version runtime is byte-identical to that commit.
• The reversible quarantine is not an upstream fix, and maintainers have not confirmed this diagnosis.
• Only the recovered session_search path was retested after containment; we did not perform broader runtime regression testing or establish result completeness.
• We are not claiming every version, platform, or storage backend is affected.

Related reports

• #3802 — the same demo/hello incompatibility makes the affected session unreadable; this report focuses on its connection to session-query reconciliation.
• #3416 — plugin-authored session event types and ignorable semantics.
• #3404 — a whole-sidebar failure from one physically malformed log; different query surface and trigger, similar isolation concern.
• #4204 — a proposed registration surface for downstream plugin session event types.

[argszero]

Verified the reported blast radius end to end against alpha.1 (cd5ef81481) — the claim holds, and I can pin the exact throw chain.

The "unsupported at sequence 3" rejection is the persistence layer, not session-query's parser. The persisted-session load path hard-rejects legacy events with an exact-seq error: session "<id>" contains unsupported legacy request/header-delta event at seq N, unsupported legacy mode/set event at seq N, and unsupported legacy request/header reason "fallback" at seq N (packages/session/session-persistence/src/coordinator.ts:281-291). A demo/hello artifact from the #3802/#3416 lineage is exactly the kind of legacy-mode payload this rejects.

The blast radius is structural and unchanged in alpha.1:

1. Every session-search query gates on _reconcile() first (packages/session-query/session-query-sqlite/src/index.ts:265 and :293).
2. _reconcile → _observeStable loads every persisted artifact whose revision differs from the index and calls persistence.inspect(id) on it (index.ts:505-511) — one artifact, no isolation.
3. The coordinator's legacy-event throw propagates out of inspect() into _observeStable's catch, which wraps it as SESSION_QUERY_PERSISTENCE_FAILED (index.ts:362-363).
4. That code maps exactly to the observed message: SESSION_QUERY_PERSISTENCE_FAILED → 'session history storage is unavailable' (packages/session-query/tool-session-query/src/service-boundary.ts:72).
5. The same all-or-nothing pattern repeats in the write leg: _reconcile wraps every replacement in a single BEGIN IMMEDIATE transaction and rolls the whole thing back on any single _replacePersistedSession failure (index.ts:437-465).

So one bad artifact poisons the global index for every search call — the report's core claim is accurate.

Two nuances worth recording for the fix direction:

• Alpha.1 has already hardened one half: extractSessionEventText now treats unknown merge-extensible event types as non-searchable ('') instead of throwing (packages/session-query/session-query/src/extraction.ts:40-41). So a purely unknown event type no longer aborts document building. But legacy request/header-delta / mode/set events still hard-reject at persistence load, and the structural all-or-nothing reconcile gate remains — so this repro class still fails on alpha.1.
• The general pattern (one bad artifact → global unavailability) is the same family as the one-bad-persisted-session DoS amplifier documented in earlier session-corruption threads — the durable fix is per-artifact isolation in reconciliation (skip-and-flag the failing artifact, degrade that session only) rather than aborting the whole index.

Thanks for the precise env details — the PRAGMA quick_check-ok-with-missing-rows distinction is a useful diagnostic tell for this failure mode.

[denial123789]

The source review above confirms the throw chain. One operational distinction is worth keeping explicit: session history storage is unavailable is too generic to justify quarantine by itself.

Before moving an artifact, join all four facts:

1. the SQLite file opens and PRAGMA quick_check is ok;
2. persisted artifact count exceeds indexed Session count;
3. supported inspection identifies one exact artifact with SessionFormatUnsupportedError, including type and sequence;
4. after stopping every writer, moving only that hash-frozen artifact outside the active scan root restores a bounded search.

Keep the quarantine reversible and outside the active root; do not edit or delete the log. Record that search coverage is now partial.

For an upstream fix, catching only SessionFormatUnsupportedError per artifact is necessary but not sufficient: stale indexed rows for that identity must be removed or tombstoned atomically, callers must receive an explicit degraded-corpus signal, and cancellation/database/identity-conflict errors must retain their typed failure paths. A later compatible reader should be able to re-admit the unchanged artifact.

I added the operator diagnosis, containment sequence, and regression gates to the independent custom-event compatibility runbook: https://github.com/sandbaseai/deepseek-harness-handbook/blob/main/docs/en/plugin-development/custom-session-events.md#one-incompatible-artifact-can-block-corpus-search

[yangfei222666-9]

Adding a consolidated, evidence-backed version of my earlier reply so the operational sequence and the claim boundaries are in one place.

Thanks to both of you for the careful source review and operational guidance. The alpha.1 (cd5ef81481) source trace identifies the same downstream failure shape we observed on our installed mixed-prerelease runtime: a persistence inspect() failure was wrapped during SQLite observation as SESSION_QUERY_PERSISTENCE_FAILED, then mapped to the service-boundary message. We have not run a clean alpha.1 end-to-end reproduction locally.

What we actually observed was lane-specific. Every session-search call we tested returned session history storage is unavailable, while the loopback HTTP endpoint and the independent ExpMem archive search remained responsive. JSONL persistence was registered, and the live backend had lazily opened the query SQLite database plus its WAL/SHM files. The same two artifact failures then reproduced deterministically in an isolated read-only inspection, while both artifacts retained valid headers, versions, cwd, and path identities. For this incident, that ruled out a missing persistence provider, a current-startup registration failure, and missing restore identity metadata; it does not identify who or what originally wrote demo/hello.

The containment decision joined the four facts you described:

1. PRAGMA quick_check was ok.
2. There were 33 persisted artifacts on disk, while SQLite persisted_sessions held 2 rows.
3. Read-only inspect() passed 31 artifacts and localized exactly two failures to SessionFormatUnsupportedError on demo/hello at sequence 3. Their frozen artifact hashes began 12d85f5a… and 804c3e4c….
4. We sent SIGTERM only to the exact desktop-owned DSH backend child; the desktop parent and launchd were not touched. We then moved only those two session directories outside the active scan root, recorded the move in a manifest, and observed the active count change only from 33 to 31. Their hashes were unchanged after the move; neither artifact was deleted or rewritten.

The desktop supervisor then restored a backend child. A later live session_search was reported by the operator as successful after the enumeration accounting was corrected. The original automated Gate D/E run had stopped on an enumeration assertion, so I am not relabeling that executor run PASS. The retained claim is only that one later operator-reported live search succeeded; it does not establish result completeness, absence of stale indexed rows, broader runtime readiness, or an upstream fix.

The quarantine remains reversible. Re-admission by a future compatible reader is a desirable compatibility property, but we have not tested it. At alpha.1, the static source shows that extractSessionEventText returns '' for merge-extensible unknown events, while specific legacy guards remain for request/header-delta, mode/set, and request/header with reason: "fallback". The structural all-or-nothing reconciliation behavior also remains, including a single BEGIN IMMEDIATE write transaction that rolls back on a replacement failure. Per-artifact isolation with an explicit degraded-corpus signal still looks like a sound direction, provided stale rows are handled atomically and cancellation, database, and identity-conflict errors retain their typed paths.

Thanks also for adding the diagnosis and containment sequence to the compatibility runbook. If useful, we can separately prepare a minimal synthetic reproduction or reviewed repro steps before sharing anything.