You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Replace SearchHistory / ReadHistory (#2679) with Recall, a retrieval layer over three kinds of Session content. No new tables, no schema migration, no write-path hook.
Measurements below come from one local workspace (6 Sessions, 1265 messages, 1.78 MB of record_json) and a synthetic 178 MB corpus built from it (~100 Sessions, the current MAX_SESSIONS_SCANNED ceiling). One data point, not a benchmark.
Motivation
runThreadSearch stops as soon as it has collected limit results, scanning Sessions newest-first. Searching 模型 in the sample workspace produces 34 real hits across 3 Sessions (29 / 4 / 1); the current algorithm returns 5, four from the largest Session, and never reaches the Session that actually discusses model context windows.
A complete scan of that workspace — no early exit, full redaction — takes 22 ms. The early exit was a reasonable assumption when the engine was written for the Desktop search box, two months before messages moved into SQLite (#1994). That assumption no longer holds, so the result set is shaped by scan order rather than relevance.
Tools
Recall(terms, question?) one call, usually enough
RecallMore(anchor) extend one passage
RecallMaterial(id) fetch an attachment's bytes (~1500 tokens for a screenshot)
Parameter sets are disjoint, budgets come from different resources (passage bytes / passage bytes / provider image bytes plus vision gating), and failure modes differ. Separate tools also stay separately grantable, since subagent allowlists match by name.
Recall returns one envelope: facts, passages, materials (metadata only), and gaps stating how far the search reached, so an empty result is distinguishable from an unsearched corpus.
The distilled layer is currently write-only. memory.sqlite holds the full extraction pipeline and eight tables, but only InteractiveLongTermMemoryWriter is wired into the Host; nothing reads those facts back into a model's context.
Attachments are unreachable today: collectSearchableText never reads attachments, and clipboard images are all named image.png, so a user message of 你看看 plus a screenshot carries no searchable trace of the screenshot at all.
Retrieval
SQL names the Sessions worth reading; JS decides what matches.
Narrowing stops at the Session, not the message. A transcript is projected
as a whole — RuntimeReadModel folds an invocation's events into messages —
so a store can vouch for "this Session could hold a match" but not for "this
message matches". Recall reads each named Session through the same readMessages a full scan uses and runs the real predicate on every message,
which is what keeps the two paths identical.
Two stores, because a transcript can live in two places. A Session born on
the ledger keeps its turns in runtime_events; one written before the ledger
keeps rows in session_messages until a read converts it. Each store scans
only the corpus it holds and the union is the candidate set. Either store
missing means part of the corpus cannot be vouched for, so the whole fast path
declines rather than answering for half of it.
A message's visible text is a JSON string value of the events that project it,
so a term inside the projected text is inside the payload — which is what makes
a payload scan a sound superset. The pre-ledger scan keeps the two reads it
always needed: an inline record is matched directly, and a record above the
chunk threshold is reassembled from its chunks first.
-- Sessions the ledger ownsSELECT DISTINCT session_id
FROM runtime_events
WHERE session_id IN (...eligible...)
AND ( instr(lower(payload_json), ?) >0OR ...
OR maka_recall_fold_unstable(payload_json) )
UNION-- a Session mid-stream: its arriving text lives in segments a per-row-- `instr` could straddle, and the read model presents it as settledSELECT DISTINCT session_id
FROM runtime_partial_snapshots
WHERE session_id IN (...eligible...)
-- Sessions still awaiting conversion, inline recordsSELECT DISTINCTmessage.session_idFROM session_messages AS message
LEFT JOIN session_message_payloads AS payload
ONpayload.session_id=message.session_idANDpayload.sequence=message.sequenceWHEREmessage.session_idIN (...eligible, ledger version <>1...)
ANDmessage.message_typeIN ('user','assistant','tool_call','tool_result')
ANDpayload.sequence IS NULLAND ( instr(lower(message.record_json), ?) >0OR ...
OR maka_recall_fold_unstable(message.record_json) )
UNION-- ... and the same over bodies reassembled from `session_message_chunks`
The invariant is SQL_candidates ⊇ true_matches. Over-selection is free — JS rejects it. Under-selection is a silent loss, so each condition that breaks the invariant has a fallback:
A term containing ", \, or a control character appears escaped in record_json: fall back to a full scan.
Messages over 64 KiB keep a marker in record_json with the body in session_message_chunks. SQLite's byte-level concatenation reassembles UTF-8 characters split by a chunk boundary, so group_concat over ordered chunks restores a searchable body. ORDER BY belongs inside the aggregate; a subquery's ORDER BY happens to work but is not guaranteed.
Recall folds text as NFC + Unicode lowercase; SQLite's lower() folds ASCII only. Terms reach the scan already folded and are matched against lower(record_json). A function registered on the connection, maka_recall_fold_unstable, flags every record where the two folds disagree — a cased letter outside ASCII, a compatibility character, a decomposed sequence — and such records are offered unconditionally. On every other record instr(lower(x), t) is exactly the predicate, so the scan never under-selects whatever script the transcript is in. Cost on a 1.8 MB workspace: the fast path goes from 7.6 ms to 17 ms per query against a 30 ms full scan.
Redaction rewrites text — it inserts [redacted], and re-serializes a JSON body it changed — so a term can occur in the redacted projection with no preimage in the stored record. Verification therefore requires a term to occur in the stored text and to survive redaction. A term found only in a redaction artifact names nothing that was said, so refusing it loses nothing.
Verification runs the same predicate on both paths, so they compare row for row:
fold(extract(m)) contains fold(q) AND fold(redact(extract(m))) contains fold(q)
The second clause is the security boundary — a credential-shaped term can never match. The first is what makes a scan of stored records a sound superset of the matches.
Measured against the ledger, with identical result sets on every query:
200 Sessions x 10 turns, 3 of them matching
a term that matches full 188 ms -> narrowed 15 ms 12.3x
a term that matches none full 187 ms -> narrowed 16 ms 12.0x
6 Sessions (a real workspace), 5 of them matching most queries
six queries full 39-43 ms -> narrowed 43-48 ms a loss
Narrowing pays exactly when most Sessions do not match, and costs a few
milliseconds when most do. The earlier "10–16x on a 178 MB corpus" figure was
measured against session_messages and no longer describes the corpus recall
reads; it is superseded by the table above.
模型 2421 ms -> 207 ms reads 2.8% of rows
图片 2092 ms -> 219 ms reads 2.8% of rows
context window 2113 ms -> 158 ms reads 0.2% of rows
zzzznotfound 2118 ms -> 135 ms reads 0 rows
It also removes a per-Session RPC round trip on the Desktop path, which today calls openSession -> loadTranscript -> close for every candidate Session and ships whole transcripts across the process boundary.
Scoring
Standard BM25, k1 = 1.2, b = 0.75. N is the corpus-wide count of searchable messages — not the candidate count, or df collapses to N and idf goes to zero for single-term queries. df comes from the verified candidate set, which is exact because candidates are a superset.
Length normalization does most of this work, but a document-type weight is still needed for a shape BM25 cannot express. Tool results are stored as JSON.stringify(content), and machine output routinely enumerates keywords so that it can be matched — a skill catalog entry in one real workspace carries eleven occurrences of a single term inside one trigger-phrase declaration. High term frequency there is evidence that the record is an index entry rather than an answer, which is the opposite of what BM25 assumes. The weight also cannot be replaced by length normalization: b sits in the denominator added to the term frequency, so its influence vanishes as that frequency grows.
Ablation on that workspace ranks the signals — tf saturation > length normalization > per-Session quota > tool-result weight — with the weight changing the ranking in two of five queries. It patches the projection rather than fixing it; the root cause is open question 3 below.
One product constraint sits outside the score, because BM25 ranks documents independently and cannot express it: at most limit/3 passages per Session, as an upper bound with a second pass filling unused slots. Without it the largest Session takes most of the result set — not because scoring is wrong, but because it genuinely contains more discussion.
Passages
A passage is the verified hit plus a bounded window of visible neighbours in the same turn, not the whole turn. In an agent Session a single turn can run to ~50 messages; one 继续做吧 produced twenty-plus consecutive terminal outputs in the sample workspace. Multiple hits in one turn collapse into one passage, and a tool_result enters a passage only as the anchor, hard-truncated.
Materials
Attachments and image tool results are aggregated per turn and returned as metadata only. RecallMaterial(id) returns { kind: 'image', mimeType, ref }, which ai-sdk-message-projection.ts already converts into a provider image part — the same path Read uses for image files. The model decides whether a screenshot is worth ~1500 tokens after reading the passage around it.
RecallMaterial never accepts a caller-supplied path. An id resolves only after verifying the attachment appears on a message in that Session, the containment check ReadHistory already applies to message_id.
Invariants
SQL narrows, JS decides — over a corpus the store actually holds. Retrieval semantics never depend on the storage query. A candidate source may only narrow a corpus it covers: naming Sessions from a table that no longer holds the transcript would answer "no match" for live history, silently. Both stores answer, or neither does.
Redaction precedes matching. Substring matching plus a hit/no-hit signal is a prefix-extension oracle, so a credential-shaped query is rejected before any corpus is touched, and matching runs on redacted text. The SQL filter runs on raw text, but its output is never returned directly.
Cost rises by layer, and every step is the model's explicit choice.
Out of scope
A derived searchable-text table. Buys ~220 ms -> ~145 ms on the 178 MB corpus; costs a schema migration, a hook in the message write path, backfill, and a second source of truth that can drift.
FTS5. Buys ~145 ms -> ~0.5 ms, which no model or user perceives, and it needs the derived text above, inheriting every cost. It also needs CJK segmentation: unicode61 treats a run of Chinese as one token (searching 显示 misses 浮窗现在只显示Miso) and trigram cannot match two-character terms, the most common Chinese word length. A JS-side bigram layer works (1.93x index size) but makes the index store a transformed form that write and query paths must agree on permanently.
Vector / semantic retrieval. Needs an embedding provider, recurring cost, offline behaviour, and a privacy decision. Worth revisiting with real recall failures as evidence.
Captioning images at ingest. A vision call per attachment on the write path.
LIKE.% and _ are wildcards; searching 100% matched 6900 rows where instr matched 200.
Revival condition for the first two: a complete scan stably exceeding 300 ms, roughly 25 MB of transcript. They should land together — once derived text is maintained, FTS5 is nearly free and replaces the hand-written scorer with bm25().
Code shape
Area
Change
packages/core
new recall pipeline, reusing the existing projection / redaction / folding operators
packages/runtime
new tool module; SessionManager exposes candidate lookup
packages/storage
two read methods on the SQLite runtime store (ledger) and two on the session store (pre-ledger), no schema change
packages/runtime-host
wire a long-term-memory reader; swap the tool composition; new protocol operation for Desktop (epoch bump)
apps/desktop
search modal adopts candidate filtering, keeps its own semantics
image projection
unchanged
Separating the Desktop search box from the Agent's retrieval is deliberate: a person wants grep, a model wants recall. Sharing one engine has tied both to the weaker contract.
Sequence
Wire the long-term-memory reader into the Host.
Multi-term matching, BM25, per-Session quota, passage assembly. Results are expected to change; a fixture query set pins the new behaviour.
SQL candidate filtering. Results must be identical to step 2, row for row — this comparison is the safety net for the whole change.
Distilled-layer density. The sample workspace holds one extracted fact, and a low-value one. Unless extraction is tuned alongside, the first layer is an empty shell and this degrades into better transcript search. The question I would most like input on.
ToolCallMessage.intent is never written. All 299 tool calls in the sample workspace lack the field and no runtime code sets it, so one of the four advertised match kinds is empty in practice. Keep it, or index visible parts of args (a shell command line, a file path)?
tool_result is 86% of searchable bytes and is stored as JSON.stringify(content), carrying escaping and structural noise into passages. Changing the projection breaks the step-3 comparison, so it needs its own change.
Archived tool results keep only a placeholder in the transcript; their bodies live in artifact storage, unreachable from any of this. Pre-existing, but worth naming.
Feedback welcome on any of it, particularly the first open question and whether retiring the two existing tools is right versus keeping them as a lower-level escape hatch.
-- ledger 持有的会话SELECT DISTINCT session_id
FROM runtime_events
WHERE session_id IN (...eligible...)
AND ( instr(lower(payload_json), ?) >0OR ...
OR maka_recall_fold_unstable(payload_json) )
UNION-- 正在流式输出的会话:到达中的文本存在分段里,逐行 `instr` 可能跨段漏掉,-- 而读模型会把它当作已定稿呈现SELECT DISTINCT session_id
FROM runtime_partial_snapshots
WHERE session_id IN (...eligible...)
-- 尚未转换的会话,内联记录SELECT DISTINCTmessage.session_idFROM session_messages AS message
LEFT JOIN session_message_payloads AS payload
ONpayload.session_id=message.session_idANDpayload.sequence=message.sequenceWHEREmessage.session_idIN (...eligible, ledger version <>1...)
ANDmessage.message_typeIN ('user','assistant','tool_call','tool_result')
ANDpayload.sequence IS NULLAND ( instr(lower(message.record_json), ?) >0OR ...
OR maka_recall_fold_unstable(message.record_json) )
UNION-- ...以及对 `session_message_chunks` 重组出的正文做同样的匹配
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Proposal
Replace
SearchHistory/ReadHistory(#2679) with Recall, a retrieval layer over three kinds of Session content. No new tables, no schema migration, no write-path hook.Measurements below come from one local workspace (6 Sessions, 1265 messages, 1.78 MB of
record_json) and a synthetic 178 MB corpus built from it (~100 Sessions, the currentMAX_SESSIONS_SCANNEDceiling). One data point, not a benchmark.Motivation
runThreadSearchstops as soon as it has collectedlimitresults, scanning Sessions newest-first. Searching模型in the sample workspace produces 34 real hits across 3 Sessions (29 / 4 / 1); the current algorithm returns 5, four from the largest Session, and never reaches the Session that actually discusses model context windows.A complete scan of that workspace — no early exit, full redaction — takes 22 ms. The early exit was a reasonable assumption when the engine was written for the Desktop search box, two months before messages moved into SQLite (#1994). That assumption no longer holds, so the result set is shaped by scan order rather than relevance.
Tools
Parameter sets are disjoint, budgets come from different resources (passage bytes / passage bytes / provider image bytes plus vision gating), and failure modes differ. Separate tools also stay separately grantable, since subagent allowlists match by name.
Corpora
memory_itemsruntime_events(ledger),session_messages(pre-ledger)Recallreturns one envelope:facts,passages,materials(metadata only), andgapsstating how far the search reached, so an empty result is distinguishable from an unsearched corpus.The distilled layer is currently write-only.
memory.sqliteholds the full extraction pipeline and eight tables, but onlyInteractiveLongTermMemoryWriteris wired into the Host; nothing reads those facts back into a model's context.Attachments are unreachable today:
collectSearchableTextnever readsattachments, and clipboard images are all namedimage.png, so a user message of你看看plus a screenshot carries no searchable trace of the screenshot at all.Retrieval
SQL names the Sessions worth reading; JS decides what matches.
Narrowing stops at the Session, not the message. A transcript is projected
as a whole —
RuntimeReadModelfolds an invocation's events into messages —so a store can vouch for "this Session could hold a match" but not for "this
message matches". Recall reads each named Session through the same
readMessagesa full scan uses and runs the real predicate on every message,which is what keeps the two paths identical.
Two stores, because a transcript can live in two places. A Session born on
the ledger keeps its turns in
runtime_events; one written before the ledgerkeeps rows in
session_messagesuntil a read converts it. Each store scansonly the corpus it holds and the union is the candidate set. Either store
missing means part of the corpus cannot be vouched for, so the whole fast path
declines rather than answering for half of it.
A message's visible text is a JSON string value of the events that project it,
so a term inside the projected text is inside the payload — which is what makes
a payload scan a sound superset. The pre-ledger scan keeps the two reads it
always needed: an inline record is matched directly, and a record above the
chunk threshold is reassembled from its chunks first.
The invariant is
SQL_candidates ⊇ true_matches. Over-selection is free — JS rejects it. Under-selection is a silent loss, so each condition that breaks the invariant has a fallback:",\, or a control character appears escaped inrecord_json: fall back to a full scan.record_jsonwith the body insession_message_chunks. SQLite's byte-level concatenation reassembles UTF-8 characters split by a chunk boundary, sogroup_concatover ordered chunks restores a searchable body.ORDER BYbelongs inside the aggregate; a subquery'sORDER BYhappens to work but is not guaranteed.lower()folds ASCII only. Terms reach the scan already folded and are matched againstlower(record_json). A function registered on the connection,maka_recall_fold_unstable, flags every record where the two folds disagree — a cased letter outside ASCII, a compatibility character, a decomposed sequence — and such records are offered unconditionally. On every other recordinstr(lower(x), t)is exactly the predicate, so the scan never under-selects whatever script the transcript is in. Cost on a 1.8 MB workspace: the fast path goes from 7.6 ms to 17 ms per query against a 30 ms full scan.[redacted], and re-serializes a JSON body it changed — so a term can occur in the redacted projection with no preimage in the stored record. Verification therefore requires a term to occur in the stored text and to survive redaction. A term found only in a redaction artifact names nothing that was said, so refusing it loses nothing.Verification runs the same predicate on both paths, so they compare row for row:
The second clause is the security boundary — a credential-shaped term can never match. The first is what makes a scan of stored records a sound superset of the matches.
Measured against the ledger, with identical result sets on every query:
Narrowing pays exactly when most Sessions do not match, and costs a few
milliseconds when most do. The earlier "10–16x on a 178 MB corpus" figure was
measured against
session_messagesand no longer describes the corpus recallreads; it is superseded by the table above.
It also removes a per-Session RPC round trip on the Desktop path, which today calls
openSession -> loadTranscript -> closefor every candidate Session and ships whole transcripts across the process boundary.Scoring
Standard BM25,
k1 = 1.2,b = 0.75.Nis the corpus-wide count of searchable messages — not the candidate count, ordfcollapses toNand idf goes to zero for single-term queries.dfcomes from the verified candidate set, which is exact because candidates are a superset.Length normalization does most of this work, but a document-type weight is still needed for a shape BM25 cannot express. Tool results are stored as
JSON.stringify(content), and machine output routinely enumerates keywords so that it can be matched — a skill catalog entry in one real workspace carries eleven occurrences of a single term inside one trigger-phrase declaration. High term frequency there is evidence that the record is an index entry rather than an answer, which is the opposite of what BM25 assumes. The weight also cannot be replaced by length normalization:bsits in the denominator added to the term frequency, so its influence vanishes as that frequency grows.Ablation on that workspace ranks the signals — tf saturation > length normalization > per-Session quota > tool-result weight — with the weight changing the ranking in two of five queries. It patches the projection rather than fixing it; the root cause is open question 3 below.
One product constraint sits outside the score, because BM25 ranks documents independently and cannot express it: at most
limit/3passages per Session, as an upper bound with a second pass filling unused slots. Without it the largest Session takes most of the result set — not because scoring is wrong, but because it genuinely contains more discussion.Passages
A passage is the verified hit plus a bounded window of visible neighbours in the same turn, not the whole turn. In an agent Session a single turn can run to ~50 messages; one
继续做吧produced twenty-plus consecutive terminal outputs in the sample workspace. Multiple hits in one turn collapse into one passage, and atool_resultenters a passage only as the anchor, hard-truncated.Materials
Attachments and image tool results are aggregated per turn and returned as metadata only.
RecallMaterial(id)returns{ kind: 'image', mimeType, ref }, whichai-sdk-message-projection.tsalready converts into a provider image part — the same pathReaduses for image files. The model decides whether a screenshot is worth ~1500 tokens after reading the passage around it.RecallMaterialnever accepts a caller-supplied path. An id resolves only after verifying the attachment appears on a message in that Session, the containment checkReadHistoryalready applies tomessage_id.Invariants
Out of scope
unicode61treats a run of Chinese as one token (searching 显示 misses 浮窗现在只显示Miso) andtrigramcannot match two-character terms, the most common Chinese word length. A JS-side bigram layer works (1.93x index size) but makes the index store a transformed form that write and query paths must agree on permanently.LIKE.%and_are wildcards; searching100%matched 6900 rows whereinstrmatched 200.Revival condition for the first two: a complete scan stably exceeding 300 ms, roughly 25 MB of transcript. They should land together — once derived text is maintained, FTS5 is nearly free and replaces the hand-written scorer with
bm25().Code shape
packages/corepackages/runtimeSessionManagerexposes candidate lookuppackages/storagepackages/runtime-hostapps/desktopSeparating the Desktop search box from the Agent's retrieval is deliberate: a person wants grep, a model wants recall. Sharing one engine has tied both to the weaker contract.
Sequence
Recalltool surface.RecallMaterial.SearchHistory/ReadHistory; Desktop adopts candidate filtering.Open questions
ToolCallMessage.intentis never written. All 299 tool calls in the sample workspace lack the field and no runtime code sets it, so one of the four advertised match kinds is empty in practice. Keep it, or index visible parts ofargs(a shell command line, a file path)?tool_resultis 86% of searchable bytes and is stored asJSON.stringify(content), carrying escaping and structural noise into passages. Changing the projection breaks the step-3 comparison, so it needs its own change.Feedback welcome on any of it, particularly the first open question and whether retiring the two existing tools is right versus keeping them as a lower-level escape hatch.
中文版
方案
用 Recall 替换
SearchHistory/ReadHistory(#2679)——一个覆盖三类会话内容的召回层。不新增表、不改 schema、不在写路径挂钩。下文的测量来自一个本地工作区(6 个会话、1265 条消息、1.78 MB
record_json),以及由它构造的 178 MB 合成语料(约 100 个会话,即当前MAX_SESSIONS_SCANNED上限)。是一个数据点,不是 benchmark。动机
runThreadSearch按会话时间倒序扫描,凑够limit条就停。在样本工作区里搜「模型」,全库真实命中 34 条、分布在 3 个会话(29 / 4 / 1);现在的算法返回 5 条,其中 4 条来自最大的那个会话,而真正讨论模型上下文窗口的会话根本没被扫到。该工作区完整扫描一遍——不早停、全量脱敏——是 22 ms。早停在当初是合理假设:这个引擎是为桌面搜索框写的,比消息迁入 SQLite(#1994)早两个月。该假设已不再成立,于是结果集是被扫描顺序决定的,而不是被相关性决定的。
三个工具
三者参数集互不相交,预算来自不同资源(段落字节 / 段落字节 / 图片字节加视觉能力门控),失败模式也不同。拆开还能分别授权——子 agent 的工具白名单是按名字匹配的。
三层语料
memory_itemsruntime_events(ledger)、session_messages(ledger 之前)Recall返回一个信封:facts、passages、materials(仅元数据)、以及gaps——说明检索到哪儿为止,让"空结果"和"没检索过"可区分。提炼层目前是只写不读:
memory.sqlite有完整的抽取管线和八张表,但 Host 只接了InteractiveLongTermMemoryWriter,没有任何地方把这些事实读回模型上下文。附件今天完全够不着:
collectSearchableText根本不读attachments,而剪贴板图片一律叫image.png——一条「你看看」加一张截图,在历史里不留下任何可检索的痕迹。检索
SQL 决定该读哪些会话,JS 决定什么算命中。
收窄止于会话,不到消息。 transcript 是整体投影出来的——
RuntimeReadModel把一次 invocation 的事件折叠成消息——所以存储能担保"这个会话可能有命中",担保不了"这条消息命中"。Recall 用与全扫相同的readMessages读取每个被点名的会话,再对每条消息跑真正的谓词,这正是两条路径结果一致的原因。两个存储,因为 transcript 可能在两处。 ledger 原生的会话把轮次写在
runtime_events;ledger 之前写的会话把行留在session_messages,直到一次读取把它转换过去。每个存储只扫自己持有的语料,并集就是候选集。任何一个存储缺席都意味着有一部分语料无人担保,此时整条快路径直接放弃,而不是只对一半作答。一条消息的可见文本是投影出它的那些事件的 JSON 字符串值,所以投影文本里的词必定在 payload 里——这正是"扫 payload 是可靠超集"的依据。ledger 之前的扫描保留它本来就需要的两次读取:行内记录直接匹配,超过分块阈值的记录先从分块拼回。
不变量是
SQL_candidates ⊇ true_matches。多选无害——JS 会丢掉;漏选是静默错误,所以每一种会破坏该不变量的情况都要有兜底:"、\或控制字符时,它在record_json里是转义形态:退回全量扫描。record_json里只留标记,正文在session_message_chunks。SQLite 的字节级拼接能还原被块边界劈开的 UTF-8 字符,所以按序group_concat可以还原出可搜正文。ORDER BY必须写在聚合内部;写在子查询里能出正确结果,但没有保证。lower()只折叠 ASCII。查询词进扫描前已折叠,与lower(record_json)比对;连接上注册一个函数maka_recall_fold_unstable,标出两种折叠结果不一致的记录——ASCII 之外的大小写字母、兼容字符、分解序列——这些记录无条件进候选。其余记录上instr(lower(x), t)就是谓词本身,因此不论转写是什么文字,扫描都不会漏选。1.8 MB 工作区上的代价:快路径从每次 7.6 ms 变为 17 ms,全量扫描是 30 ms。[redacted],改动过的 JSON 正文会被重新序列化——所以一个词可能只出现在脱敏后的投影里,而原始记录中没有它的前像。因此判定要求词既出现在原始文本中,又在脱敏后仍然存在。只出现在脱敏痕迹里的词并不对应任何说过的话,拒绝它没有损失。两条路径跑同一个谓词,因此可以逐条对拍:
第二个子句是安全边界——凭据形态的词永远匹配不上。第一个子句让对原始记录的扫描成为匹配集的可靠超集。
在 ledger 上实测,每条查询结果集完全一致:
收窄的收益恰好出现在"多数会话不命中"时;多数会话都命中时它要多花几毫秒。此前"178 MB 语料快 10–16 倍"是对着
session_messages测的,已不描述 recall 实际读取的语料,以上表为准。它同时消掉了桌面端的逐会话 RPC 往返——今天每个候选会话都要
openSession -> loadTranscript -> close,把完整转录搬过进程边界。打分
标准 BM25,
k1 = 1.2、b = 0.75。N取全库可搜消息数,不是候选数——否则df会等于N,单词查询的 idf 归零。df从判定通过的候选集里数,因为候选是超集,所以精确。长度归一化做了大部分工作,但仍然需要一个按消息种类的权重,来表达 BM25 表达不了的一种形状。工具结果以
JSON.stringify(content)存储,而机器输出为了能被匹配到,常常成段罗列关键词——某个真实工作区里,一条技能目录记录在单条触发词声明里就包含同一个词的十一次出现。在那里,词频高恰恰说明这条记录是索引条目而不是答案,与 BM25 的假设相反。长度归一化也替代不了它:b在分母里与词频相加,词频一大,它的影响就趋近于零。该工作区上的消融给出信号强弱——tf 饱和 > 长度归一化 > 跨会话配额 > 工具结果权重——其中该权重在五个查询里改变了两个的排序。它是在给投影问题打补丁而不是修它;根因是下面的开放问题 3。
有一条产品约束落在打分之外,因为 BM25 是逐文档打分、表达不了它:单个会话最多占
limit/3条,作为上限,另有第二遍补满空缺名额。没有它,最大的会话会占走大部分结果——那不是打分错了,是它确实讨论得更多。段落
一个 passage 是"判定通过的那条消息 + 同一轮内前后有限条可见消息",不是整轮。agent 会话里一轮可能有约 50 条消息——样本工作区里一句「继续做吧」引出了二十多条连续终端输出。同一轮内的多个命中合并为一个 passage;
tool_result只有作为锚点时才进 passage,且硬截断。材料
附件和图片型工具结果按轮聚合,只返回元数据。
RecallMaterial(id)返回{ kind: 'image', mimeType, ref },ai-sdk-message-projection.ts已经会把它转成 provider 的 image part——Read读图片文件走的就是这条路。模型读完段落文本之后,自己决定这张截图值不值约 1500 token。RecallMaterial绝不接受调用方传来的路径。id 只有在验证该附件确实出现在那个会话的某条消息上之后才解析——这是ReadHistory对message_id已有的越界校验。三条不变量
不在范围内
unicode61把连续中文当成一个 token(搜「显示」匹配不到「浮窗现在只显示Miso」),trigram匹配不了双字词——而双字恰是中文最常见的词长。JS 侧做双字切分可行(索引 1.93 倍),但这会让索引存的不是原文而是一种变换后的形态,写入和查询两侧必须永久保持一致。LIKE。%和_是通配符;搜100%命中 6900 行,而instr是 200 行。前两项的复活条件:一次完整扫描稳定超过 300 ms,约相当于 25 MB 转录。它们应当一起落地——一旦派生文本的维护成本已经付了,FTS5 几乎白送,还能把手写的打分换成
bm25()。代码落点
packages/corepackages/runtimeSessionManager透出候选查询packages/storagepackages/runtime-hostapps/desktop把桌面搜索框和 Agent 的召回分开是刻意的:人要的是 grep,模型要的是 recall。共用一个引擎,等于让两者都受制于较弱的那份契约。
落地顺序
Recall工具面。RecallMaterial。SearchHistory/ReadHistory;桌面端接入候选筛选。开放问题
ToolCallMessage.intent从来没被写过。 样本工作区里 299 条工具调用全都没有这个字段,runtime 里也没有任何代码设置它——所以对外宣称的四个命中类型里,有一个在实际数据上是空的。是留着,还是改成召回args里的可见部分(shell 命令行、文件路径)?tool_result占可搜字节的 86%,且以JSON.stringify(content)存储,把转义和结构噪声带进段落。改这个投影会让第 3 步的对拍失效,因此需要单独一次改动。以上任何一点都欢迎反馈,尤其是第 1 条,以及"退役那两个现有工具"是否正确——还是应该把它们保留为更底层的逃生舱。
All reactions