perf(sessions): probe transcript message presence without payload decoding

[steipete]

What Problem This Solves

Embedded session bootstrap decodes a transcript message just to determine whether history exists. Large stored messages therefore allocate unnecessary payload strings and objects before context preparation. A synthetic 16 MiB message made twenty presence checks take 258 ms and grow RSS by 224 MiB.

Why This Change Was Made

Add a typed Kysely presence query at the session accessor boundary using the existing transcript identity index. Canonical message history is answered from index metadata. Exact SQLite imports and id-less rows can legitimately lack identities, and an identity type can be NULL, so unclassified rows retain the existing tolerant JSON parser. Both probes share a deferred SQLite snapshot so concurrent classification cannot hide an always-present message. The raw probe excludes a single set of classified sequences; this avoids SQLite choosing a quadratic per-event identity join. Physical history includes messages before reset or on older branches.

The persistence owner answers the domain query, keeping caller-side payload inspection out of future database adapters. SQLite schema and indexes remain unchanged.

User Impact

Existing embedded sessions avoid decoding stored message payloads during the bootstrap presence check. Detached-session precedence, CLI presence semantics, imported rows, native evidence, malformed rows, and deep JSON keep their existing behavior.

Evidence

• Regression test against the previous code decoded 4,194,373 bytes solely for a boolean; the candidate decodes zero for the same real SQLite/embedded-helper fixture.
• 182 tests passed across the accessor, exact import, embedded detached-session, and bootstrap-presence suites. After adding nullable-identity and two-connection snapshot regressions, all 23 focused bootstrap/import tests passed again.
• Synthetic twenty-check benchmark over a 16 MiB message: 258.11 ms before, 1.58 ms after; RSS growth 224 MiB before, 16 KiB after. This is a focused measurement, not a general Gateway latency claim.
• SQLite explains the canonical probe as a covering-index search using the existing idx_agent_transcript_event_sequence.
• Full changed gate passed before review follow-ups; final candidate reran core/core-test types, targeted lint, Kysely guardrails, focused tests, and P2 independent autoreview successfully.
• Production delta: +57 lines for the indexed owner operation and shared parser, tests: +164, obsolete assertion-baseline entry: -1.

Independent actual embedded-bootstrap-helper validation passed on commit 466377d80d1d589b5ca4cbc83ff2f2190eae5688: twenty reads of 16 MiB canonical history behind reset took 2.27 ms total, detached in-memory ownership remained authoritative, native evidence alone was excluded, nullable identity types were read correctly, and SQLite integrity passed. Materialized source tree 455524a787f5a197b3e8ce9e3751ed51749a3393 matched before/after.

Refreshed onto main c861706c2d8af6300a8597ebe3bd9f2b96ffcb2d to acquire the independently landed recall-fixture fix in #138123. The feature patch is unchanged. Independent native validation passed again on exact head 81b3d68253470b9f9cf323927b08f931596ee132, tree 9c14f5854d07f580e5d4b7e89ea270d31c5bac08. The actual embedded helper passed large canonical history, detached ownership, native-evidence exclusion, nullable identity fallback, and SQLite integrity. All 46 bootstrap/import/context tests, including landed eager batching, and fresh P0–P2 review passed. All synthetic processes and state were cleaned up; source stayed clean.

Final candidate verification

Final candidate fd666ae7a105b6dd49fb3354cd4a336a581e0658 is rebased onto 91aad5cfcf054c05e4c0ac78c0156d07a7a8e6bf. The feature patch is unchanged, and the persistence owners, callers, codecs, and schema contracts have no relevant intervening drift. The shared CI fixture failure was independently repaired in #138238; the canonical repair passed all 1,666 tests in the original 112-file runner configuration on the rebased cron candidate.

All eight database changes were exercised together on the final combined staged tree 8fa80406d1564bac0cc3968df20be6ee12d93dc1: 228 tests across 13 files passed. A separate native run used one synthetic state root to verify retained KV/blob facade objects across physical connection reopen, corrupt-sweep rollback, real cron command receipts across Gateway restart, stale/exact delivery acknowledgement and media custody, 1,201-message branch context/presence, transcript byte preservation through two memory reindexes, current-writer success, strict-closure refusal, and stale successor append/branch rejection. A fresh process reopened the final state and passed integrity checks. Source remained unchanged; synthetic state and Gateway processes were removed. This is local macOS runtime proof; hosted platform/security CI remains a separate required gate.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: blocked before merge. Reviewed September 4, 2026, 9:07 AM ET / 13:07 UTC.

ClawSweeper review

What this changes

The PR adds an index-backed SQLite transcript-presence query so embedded-session bootstrap can detect message history without decoding canonical message payloads.

Merge readiness

⛔ Blocked before merge - 2 items remain

Keep open for normal maintainer acceptance. Current main still decodes transcript rows for this presence check; the proposed accessor-owned index probe preserves the raw fallback for unclassified imports, and no correctness or security defect was found. Automatic cleanup is not appropriate because the PR is marked maintainer and Repository State records a MEMBER author.

Priority: P3
Reviewed head: fd666ae7a105b6dd49fb3354cd4a336a581e0658
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	A focused owner-boundary change with relevant regression coverage and no supported correctness finding.
Proof confidence	🌊 off-meta tidepool	Not applicable: Repository State records a MEMBER author, so the external-contributor proof gate does not apply; the PR body nevertheless supplies an exact-head native embedded-bootstrap validation claim tied to this accessor path.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: Repository State records a MEMBER author, so the external-contributor proof gate does not apply; the PR body nevertheless supplies an exact-head native embedded-bootstrap validation claim tied to this accessor path.
Evidence reviewed	7 items	Current main behavior: Current main still implements the bootstrap predicate by enumerating transcript rows and JSON-decoding each candidate until a message event is found; the requested performance improvement is therefore not already implemented. Accessor-owned implementation: The introduced operation first probes the existing message identity index, then decodes only rows lacking a classified identity inside one deferred SQLite snapshot. Caller boundary: Embedded bootstrap delegates durable transcript presence to the session accessor while retaining the supplied in-memory SessionManager as authoritative for detached sessions.
Findings	None	None.
Security	None	None.

How this fits together

Embedded agent bootstrap determines whether persisted conversation history exists before invoking context-engine setup. The session accessor reads the agent SQLite transcript store and returns that presence fact to the embedded runner.

flowchart LR
A[Embedded session bootstrap] --> B[Transcript presence request]
B --> C[Session SQLite accessor]
C --> D{Identity index has message}
D -->|Yes| E[Return presence without decoding]
D -->|No or unclassified| F[Inspect unclassified raw rows]
E --> G[Context-engine bootstrap]
F --> G

Loading

Decision needed

Question	Recommendation
Should the session/persistence owner approve the new index-backed transcript-presence operation for existing SQLite transcripts?	Approve the current implementation: Accept the indexed fast path because it preserves detached-manager ownership and tolerant handling of unclassified transcript rows.

Why: The implementation is focused and no defect was found, but this maintainer-labeled PR is not eligible for automated closure or final acceptance.

Before merge

• Complete next step (P2) - Obtain maintainer approval for this maintainer-authored SQLite/session performance change.
• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Agent review details

Security

None.

PR surface

Source +57, Tests +164, Other -1. Total +220 across 7 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	4	74	17	+57
Tests	2	165	1	+164
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	1	0	1	-1
Total	7	239	19	+220

Review metrics

Metric	Value	Why it matters
Production versus test delta	production +57, tests +164, baseline −1	The production growth is justified by moving the boolean query to the persistence owner and is covered by focused canonical, import, malformed-row, and snapshot cases.

Technical review

Best possible solution:

Retain the accessor-owned indexed probe and raw fallback, then have the session/persistence owner approve the focused performance change on its exact green head.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR improves an internal persistence read path rather than reporting a user-facing failure. The added test provides a concrete source-level performance scenario with a large canonical transcript row.

Is this the best way to solve the issue?

Yes. The session accessor owns the SQLite identity index and durable transcript reads, while the embedded runner remains responsible only for choosing the supplied in-memory manager when one exists.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a220f55bbbe1.

Labels

Label changes:

• add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• remove rating: 🦞 diamond lobster: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

• P3: This is a bounded internal performance improvement with no demonstrated change to user-visible session behavior.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Repository State records a MEMBER author, so the external-contributor proof gate does not apply; the PR body nevertheless supplies an exact-head native embedded-bootstrap validation claim tied to this accessor path.

Evidence

What I checked:

• Current main behavior: Current main still implements the bootstrap predicate by enumerating transcript rows and JSON-decoding each candidate until a message event is found; the requested performance improvement is therefore not already implemented. (src/config/sessions/session-accessor.sqlite-read.ts:464, a220f55bbbe1)
• Accessor-owned implementation: The introduced operation first probes the existing message identity index, then decodes only rows lacking a classified identity inside one deferred SQLite snapshot. (src/config/sessions/session-accessor.sqlite-read.ts:465, fd666ae7a105)
• Caller boundary: Embedded bootstrap delegates durable transcript presence to the session accessor while retaining the supplied in-memory SessionManager as authoritative for detached sessions. (src/agents/embedded-agent-runner/run/attempt-transcript-helpers.ts:165, fd666ae7a105)
• Storage contract: The existing identity table stores an optional event type and has a covering session/type/sequence index; this PR neither changes the schema nor adds persistent data. (src/state/openclaw-agent-schema.sql:463, fd666ae7a105)
• Focused regression coverage: The added tests exercise a large canonical message without payload parsing, an interleaved identity classification update, id-less and nullable-identity rows, malformed rows, and non-message native evidence. (src/agents/embedded-agent-runner/run/attempt-transcript-helpers.presence.test.ts:10, fd666ae7a105)
• Verified merge surface: The verified GitHub test merge changes the same seven files as the pinned introduced delta, with no base-only files attributed to this PR. (aab89fd7da74)

Likely related people:

• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (3 earlier review cycles)

• reviewed 2026-09-04T10:55:20.724Z sha 466377d :: needs changes before merge. :: none
• reviewed 2026-09-04T11:15:08.935Z sha 81b3d68 :: needs maintainer review before merge. :: none
• reviewed 2026-09-04T11:54:13.176Z sha 2e71ca3 :: blocked before merge. :: none

[sylvesterkaczmarek]

The persistence owner is the right place for this presence query. Sharing one SQLite snapshot across indexed and raw fallback probes is important; otherwise concurrent identity classification could turn a physically present message into a false negative.

[steipete]

Merged via squash.

• Prepared head SHA: fd666ae7a105b6dd49fb3354cd4a336a581e0658
• Landed commit: 13b8f1de684e6e197876fc3c77489f03f10a89f3