Replies: 1 comment
|
Two field samples from PR #4879, found by reviewer @likun666661. Both sit in code added by the PR itself—new code written after the rules were published, by someone who had read them. That is the point worth reporting: this is not legacy drift. Bounded Read — the budget was checked after the read it was meant to refuse.
At the call site, bounded and unbounded shapes read almost identically. The only difference is whether the check sits before or after materialization, and neither the type system nor the review surface shows that. The unbounded readers already listed in the discussion are at least legible as unbounded; this one is not. Measure once — the length was measured in the wrong unit. The same guard accumulated This is the inverse of rule 4's stated failure mode. The rule warns about re-encoding to recover a known length, whereas here a length crossed a boundary and its unit silently changed. Same root cause: a number crossed a layer without its unit. And one that is not fixed, as the rule's own example.
Two measurements, and the second is the one that matters:
At 2,000 rounds a page takes about a second, and nothing caps that as the Session keeps growing. (Synthetic data, temp SQLite, Node 25.6.1 — not production latency. The I first described the consequence here as bounded latency. That was wrong, and it is worth correcting rather than quietly dropping: a per-page cost with no ceiling is what this discussion means by unbounded, and scrolling back through a long Session is the path a reader actually takes. Not fixed in #4879 because removing the scan needs a persisted per-invocation ordinal range: a new table, a migration, and consistency maintenance on the append path—its own change, not a line in a refactor. Tracked in this discussion rather than a parallel issue, because this is where the rule lives. This hits two of the discussion's own near-misses at once: "A sessionId is not a bound. One session also grows without limit." and "LIMIT bounds returned rows, not scan, sort, or large-field decode." The table above lists 中文PR #4879 评审中,@likun666661 发现了两处现场样本。两处都在该 PR 自身新增的代码里——这正是最值得指出的地方:它们不是历史遗留代码的漂移,而是在规则发布之后,由读过规则的人写出的新代码。 Bounded Read —— 预算检查落在了本该拦截的读取之后
在调用点,有界与无界读取几乎没有视觉差异,区别仅在于检查发生在物化之前还是之后,类型系统和 Review 界面对此都完全静默。讨论帖中列出的那些无界读取,至少一眼就能辨认出是无界的;但这一处不是。 Measure once —— 长度度量用错了单位 同一处防护逻辑累加的是 这正好是规则 4 所述失效模式的镜像反例。规则警告的是为获取已知长度而重复编码;这里则是长度跨层传递后,单位含义被悄然改变。两者的根因完全一致:一个数字跨越层次时,丢掉了它携带的单位。 还有一个未修的样本,作为规则自身的实例
两组实测,真正要紧的是第二组:
2,000 轮时每翻一页约 1 秒,而随着 Session 继续增长,这个值没有任何上限。(合成数据、临时 SQLite、Node 25.6.1,不是生产延迟。 我最初把这里的后果写成「延迟有界」。那是错的,而且值得明说而不是悄悄改掉:没有上限的单页成本,正是本讨论所说的「无界」;而往回滚一个长会话,恰恰是读者真实会走的路径。 未在 #4879 中修复,是因为消除全量扫描需要持久化每个 invocation 的 ordinal 范围:这意味着需要新表、迁移以及追加写入链路的一致性维护。这是一次独立改动,不是重构 PR 里顺手改一行。在此处跟踪而非另建 Issue,是因为规则的源头就在这里。这一处同时命中了本讨论的两条近似陷阱:"A sessionId is not a bound. One session also grows without limit." 与 "LIMIT bounds returned rows, not scan, sort, or large-field decode." 讨论帖上方的对照表将 |
Uh oh!
There was an error while loading. Please reload this page.
Our docs and blog posts define a read/write architecture. An audit across
packages/storage,packages/runtime-host,packages/runtime, andapps/desktopfound that each rule holds in exactly one place and almost nowhere else — roughly thirty sites in total.This is not a proposal; nothing here needs a vote. It is a heads-up on what we found, what we are fixing, and one question that is a product decision rather than a storage one.
The rules and where they come from
1. Bounded Read.
docs/blogs/log-is-the-runtime.md:192:docs/architecture/runtime-host-architecture.md:67says the same for the wire:Every collection that grows with use requires this, not just context offload and the protocol. A read is a keyed lookup, a bounded page or increment, or a full traversal entered by name. The budget applies before rows are materialized or decoded. Responses state whether they are complete; a bounded read never reports truncation as absence. Explicit full traversals (recovery, backup, export, migration) must state their watermark and resume behavior.
Four ways to satisfy the letter and miss the point:
sessionIdis not a bound. One session also grows without limit.LIMITbounds returned rows, not scan, sort, or large-field decode.omittedby full scan defeats the purpose;hasMoreis usually enough.2. Authority and projection are decidably different.
docs/blogs/multi-agent-scheduling.md:305:log-is-the-runtime.md:60, and what follows at:42:Re-derivable without side effects defines a projection: disposable, rebuilt rather than migrated, selecting only consumed fields. Persisted atomically once defines authority: written before projection updates, never mirrored into a long-lived in-memory copy that answers reads on the store's behalf. A durable projection records what a Compaction Checkpoint records — its range, terminating watermark, and source digest.
Clarification:
multi-agent-scheduling.mdnotes the Coordinator keeps no long-lived authoritative DAG in memory, re-extracting baseline facts on each reconciliation. It rebuilds a projection from a bounded graph snapshot. Rebuilding an authority in memory, unbounded, is the opposite pattern.3. Creation has an owner; retirement needs one too.
log-is-the-runtime.md:213names the price of append-only:We built the first two; lifecycle management exists only in
ContextOffloadStore.Any change introducing durable data or external files must define in that same change: lifetime owner, keep-alive condition, retirement trigger, and the component performing retirement. If deletion can fail, the obligation must remain discoverable and retryable. Replacing a cleanup mechanism must either cover existing data or state the retention change. Long retention is fine if an explicit lifecycle owns it. For lock files, deletion must be proven not to break mutual exclusion.
4. Measure once, carry the measurement. Encoding is a bounded read of a value. Re-encoding to recover a known length repeats the same waste one layer up. Reuse the encoding or carry a trusted length across the same value, version, and boundary. Frame, normalized object, and sub-object budgets are separate and do not substitute for one another.
Each rule is already implemented correctly, once
desktop-transcript-replica— 16 KiB bootstrap tail,anchorSequencepaging,throughSequencecatch-up;bounded-evidence.tssupplies the budget type and alimit_exceededarmsession_catalog_projection— seven columns, covering indexes, joins the payload only for the page it returnsclaude-code-session-adapter— mtime and size, caches misses, drops entries the source no longer listssqlite-context-offload-store— content-addressed, refcounted,ON DELETE RESTRICT, GC queuehost-change-feed— frames carry a kind and a revision; clients re-readagent.graph.query— slices and backfills anomittedcountsession-transcript-pager— retains the encoded buffers and measures the retained bytesIn most cases the correct implementation already exists beside the incorrect one. Nothing below needs a new abstraction.
Where we are not doing it
Counts exclude tests and generated output.
Bounded Read.
readRuntimeEvents,ModelCallLedgerReader.read, andSqliteArtifactMetadataRepository.readAllaccept no bounds.listSessionInvocationsbindslimit: options.limit ?? -1. Bounded variants are nearly unused:listSessionInvocationsBoundedhas one real caller against nineteen for the unbounded version;readEventsByTypeBoundedhas one, which runs an unbounded fetch after a bounded pre-check.listRootTurnAdmissionsForRecoverydecodes every admission so callers can test.length > 0.runtime-event-store.ts:169lists a session's invocations to.findone, beside a keyedreadRunInvocation. Thread search reads whole transcripts for up to 200 sessions to substring-match.session_catalog_projection.is_archivedleads three indexes with no query-path reader, while about twenty call sites decode the full record to checkheader.isArchived.Authority vs projection.
ArtifactStorerebuilds the entire record table into memory before every read and mutation; that mirror answers reads.Retirement. Twelve tables are created and written in production with no delete and no live cascade:
core_shell_runs,core_interaction_requests(andcore_interaction_outcomes, whoseON DELETE CASCADEcan never fire because the parent is never deleted),usage_llm_calls,usage_tool_invocations,usage_model_call_attempts,subagent_spawns,workhub_action_claims,session_metadata_tombstones,memory_items,memory_write_operations,memory_extraction_cursors, andmemory_compaction_policy_denials.context-offloadsets and validatesauto_vacuum = INCREMENTAL, butPRAGMA incremental_vacuumnever runs.ContextOffloadStore.collectGarbageis reached only when a session is retired or a conversation copy is discarded, leaving blobs unreferenced by normal evictions waiting for an unrelated event.State-root owner lock files accumulate: 94,168 zero-byte files on one developer machine. Not deleting them is deliberate and documented — deleting a cache directory must never make the same state root acquirable again. The defect is that the key space is unbounded, since
rootIdis freshly minted per storage root, so every test run and temporary root leaks one permanently. A second lock in the control directory serves the same purpose, so the fix is likely to remove one authority rather than add anunlink.Measure once.
requireEncodedByteLimitis called 51 times and defined five times — one exported, four byte-identical private copies. Every call encodes and discards. Subscription frames are encoded five or six times before reaching the socket; only the last encoding is sent. Transport computes the exact byte length on receive and drops it; the protocol decoder re-encodes the parsed value to recover the same number. Six page builders re-encode the whole accumulated page per added item.Whole-collection rewrites. Any scheduled-task mutation, including advancing one
nextFireAt, deletes both scheduled-task tables and reinserts every row. Each shell-run flush reads the record, deep-compares accumulated output, and writes the whole record back.What happens next
These are ordinary bugs addressed in ordinary PRs, grouped by intent rather than search hits — collapsing the five duplicate byte-limit helpers is one change, not five.
Two supporting pieces accompany them:
testjob, patterned aftercheck-renderer-architecture.mjs, re-deriving the base from the base commit's source tree and comparing per call edge rather than by repository-wide totals.The ledger matters more than a prohibition would. In #4808 the retired-capture sweep was removed, and the PR said plainly that the remainder is a deliberate one-time upgrade residue rather than a promise of physical deletion. Review accepted that, correctly — and then the statement disappeared from view on merge. A rule saying cleanup capability may not decrease would have flagged a sound change. A ledger gives an accepted residue somewhere to live.
An ADR follows once initial changes prove the shape, recording what was decided rather than proposing it.
Known, not acted on yet
memory_items,memory_write_operations,memory_extraction_cursors,memory_compaction_policy_denials,session_metadata_tombstones,subagent_spawns, andworkhub_action_claimshave no retirement path, but nobody has measured whether they actually grow. They go into the ledger until someone does.Appends to
runtime_partial_segmentsare already segmented at 64 KiB; the tail rewrite is bounded and may be nothing more than engine cost. Deferred until someone measures WAL volume.One open question
usage_llm_calls,usage_tool_invocations, andusage_model_call_attemptsalso lack retirement paths, but the answer is a product decision rather than a storage one: should usage records survive session deletion, and for how long?Related: #4037, #4038, #4071, #4808, and the cutover invariant stated in #4030 — "surviving metadata stores must use indexed row operations rather than
readAll/replaceAll" — exactly the kind of decision that should have had a document to live in.中文版
四条既定规则,各自仅有一处真正执行
我们在自己的文档与博客中阐述了一套读写架构。对
packages/storage、packages/runtime-host、packages/runtime和apps/desktop的审查发现,每条规则在全库中几乎都仅有一处真正成立——涉及的违规调用点总计约三十处。这不是新设计提案,也不需要投票。这是一次情况通报:列出我们发现的问题、正在修复的内容,以及一个属于产品决策而非存储决策的问题。
规则及其出处
1. 有边界读取(Bounded Read)。
docs/blogs/log-is-the-runtime.zh-CN.md:192:docs/architecture/runtime-host-architecture.md:67对通信传输层也作了相同规定:所有随使用而增长的集合都需要遵守此规则,不仅限于上下文卸载与通信协议。读取操作只能是按键查询、有边界的分页或增量读取,或是显式声明的全量遍历。开销预算必须在物化或解码行数据之前生效。响应必须标明是否完整;有边界读取绝不能将截断结果报告为不存在。显式全量遍历(恢复、备份、导出、迁移)必须声明其水位线与恢复行为。
四种表面合规但违背本质的做法:
sessionId不是边界。单个会话同样会无界增长。LIMIT限制的是返回行数,而不是扫描、排序或大字段解码。omitted数量得不偿失;通常hasMore已足够。2. 权威与投影是可判定地区分开的。
docs/blogs/multi-agent-scheduling.zh-CN.md:305:docs/blogs/log-is-the-runtime.zh-CN.md:60以及随后的:42:可无副作用反复推导即为投影:用完即弃,重建优于迁移,且仅选取消费者所需的字段。一次性原子持久化即为权威:必须在任何投影更新前写入,绝不能镜像到长期驻留内存的副本中代为响应读取。持久化投影记录的内容与 Compaction Checkpoint 相同:自身范围、终止水位线以及源数据的摘要。
澄清一点(初读容易误解):
multi-agent-scheduling.md指出 Coordinator 不在内存中维护长期驻留的权威 DAG,而是在每次对齐时重新提取基线事实。它重建的是投影,读取的是有边界图快照。在内存中无界重建权威事实,恰恰是相反的反模式。3. 创建有主,退休也必须有主。
docs/blogs/log-is-the-runtime.zh-CN.md:210:我们实现了前两点;生命周期管理仅存在于
ContextOffloadStore中。任何引入持久化数据或外部文件的改动,必须在同一次改动中明确:生命周期属主、存活条件、退役触发点,以及执行退役的组件。如果删除可能失败,该义务必须保持可发现且可重试。替换清理机制时,必须覆盖已有数据或声明保留策略变更。长保留期并无不可,只要指明负责其生命周期的机制。对于 lock 文件,还必须证明删除不会破坏互斥性。
4. 度量一次,带着走。 编码是值的有边界读取。重新编码以获取已知长度,是在上一层重复同样的浪费。同一值、版本与边界内,应复用编码结果或传递可信长度。Frame、归一化对象与子对象的预算各自独立,不可相互替代。
每条规则都已有一处正确实现
desktop-transcript-replica— 16 KiB 引导末尾数据,anchorSequence分页,throughSequence追赶;bounded-evidence.ts提供预算类型与limit_exceeded分支session_catalog_projection— seven columns,覆盖索引,仅为返回的分页 join 载荷claude-code-session-adapter— mtime 与 size,缓存未命中,移除源不再列出的条目sqlite-context-offload-store— 内容寻址,引用计数,ON DELETE RESTRICT,GC 队列host-change-feed— frame 携带 kind 与 revision;客户端重新读取agent.graph.query— 切片并回填omitted计数session-transcript-pager— 保留编码后的 buffer 并测量保留字节数大多数情况下,正确实现已与错误实现并存。以下内容均不需要引入新抽象。
哪些地方未执行
统计不包含测试与生成代码。
Bounded Read.
readRuntimeEvents、ModelCallLedgerReader.read和SqliteArtifactMetadataRepository.readAll均未设置边界。listSessionInvocations绑定的参数为limit: options.limit ?? -1。有边界版本几乎未被使用:listSessionInvocationsBounded仅有 1 处真实调用者,而无边界版本有 19 处;readEventsByTypeBounded仅有 1 处调用者,且其在有边界预检后紧接着进行无边界拉取。listRootTurnAdmissionsForRecovery解码所有 admission 仅用于调用方检查.length > 0。runtime-event-store.ts:169通过遍历会话的 invocation 列表执行.find,而旁边就存在按键查询的readRunInvocation。线程搜索为进行子串匹配,会读取多达 200 个会话的完整 transcript。session_catalog_projection.is_archived位于 3 个索引的前列,在查询路径中却无任何读取方,而约 20 处调用点为了检查header.isArchived却解码完整记录。Authority vs projection.
ArtifactStore在每次读取和每次变更前,都将整张记录表重新构建到内存中,并由该内存镜像响应读取。Retirement. 生产环境中创建并写入了 12 张表,却既无删除也无生效的级联操作:
core_shell_runs、core_interaction_requests(以及core_interaction_outcomes,其ON DELETE CASCADE因父表记录从未删除而永远不会触发)、usage_llm_calls、usage_tool_invocations、usage_model_call_attempts、subagent_spawns、workhub_action_claims、session_metadata_tombstones、memory_items、memory_write_operations、memory_extraction_cursors、memory_compaction_policy_denials。context-offload设置并校验了auto_vacuum = INCREMENTAL,但PRAGMA incremental_vacuum从未执行。ContextOffloadStore.collectGarbage仅在会话退役或丢弃 conversation copy 时才会被触发,常规淘汰解引用的 blob 只能等待无关事件发生。状态根 owner lock 文件持续堆积:在一位开发者的机器上发现了 94,168 个零字节文件。不删除它们是有意的且有文档依据——删除缓存目录绝不能让同一个 state root 重新变得可获取。缺陷在于键空间无界:
rootId为每个 storage root 全新生成,导致每次测试运行和临时 root 都会永久泄漏一个文件。控制目录中的第二把锁起着同样的作用,因此修复方案很可能是移除一处 authority,而非直接追加unlink。Measure once.
requireEncodedByteLimit被调用 51 次,定义了 5 次——1 个导出,4 个字节完全一致的私有副本。每次调用都是编码后即丢弃。一个订阅 frame 在到达 socket 前会被编码 5 或 6 次,最终只有最后一次编码被发送。传输层在接收时计算了精确字节长度随后丢弃;协议解码器为了找回同一个数字,又将解析后的值重新编码。6 个页面构建器在每追加一项时,都会对整个累积页面重新编码。Whole-collection rewrites. 任何 scheduled-task 的变更(包括推进一次
nextFireAt),都会删除两张 scheduled-task 表并重新插入所有行。每次 shell-run 刷新都会读取整条记录,深度比对累积输出,再将整条记录写回。后续处理
这些都是普通 bug,将通过常规 PR 进行修复,按意图而非按搜索结果分组——收敛 5 处重复的 byte-limit 辅助函数是一次改动,而不是五次。
伴随这些修复将引入两项配套设施:
testCI 任务中增加一项检查,结构参照check-renderer-architecture.mjs,从 base commit 的实际源码树重新推导基线,按 call edge 而非全库总量进行比对。台账比单纯的禁止规则更有价值。在 #4808 中,退役捕获清理逻辑被移除,PR 明确说明残留是故意的单次升级残留,而非物理删除承诺。评审接受了这一点,也是正确的(correctly)——但这句声明在 merge 后便不再可见。如果设置一条“清理能力不得下降”的硬性规则,就会误伤这种合理的改动。台账让这些被接受的残留有了记录之处。
在第一批改动验证模式后,会跟进一份 ADR,用于记录已确定的决策,而非进行前期提案。
已知但暂不处理
memory_items、memory_write_operations、memory_extraction_cursors、memory_compaction_policy_denials、session_metadata_tombstones、subagent_spawns和workhub_action_claims均无退役路径,但目前尚无人测量它们是否真正在增长。在有人测量前,它们先记入台账。对
runtime_partial_segments的追加已按 64 KiB 分段;尾部重写是有界的,可能纯属引擎固有开销。在有人测量 WAL 容量前,暂不处理。一个待决的产品问题
usage_llm_calls、usage_tool_invocations和usage_model_call_attempts同样缺少退役路径,但这属于产品决策而非存储决策:usage 记录在 session 删除后是否应当保留?如果保留,保留多久?相关 issue:#4037、#4038、#4071、#4808,以及 #4030 中声明的割接不变量——"surviving metadata stores must use indexed row operations rather than
readAll/replaceAll"——这正是理应有一份专门文档来承载的决策。All reactions