Replies: 2 comments
|
I ran read-only measurements on the same installation that produced the reported numbers. The real data both confirms and refines the audit. The ~47 KB attribution needs a timeline.
The slope, not just the level: since the original report (~1 day), On the questions: Q1 — an explicit evidence/replay grade, off by default. Q2 — One connection to #4666: a per-call durable byte budget is also what makes any future remote commit path viable — an ordered commit pipeline cannot carry payloads of unbounded size. 中文版本我在产出报告里那些数字的同一个环境上做了只读实测。真实数据既证实了审计结论，也需要做一点修正。 47 KB 的归因要加一条时间线。 真实环境里的 重要的是斜率，不只是存量： 原始报告之后约一天， 关于问题：Q1——完整请求重建应是显式开启的 evidence/replay 级别，默认关闭。Q2——正常 Session 有 与 #4666 的一个衔接：每次调用持久字节有界，也是将来任何远程提交路径可行的前提——有序提交管道载不动无界载荷。 |
EnglishThanks for the audit — I reproduced it independently and it holds. I want to add one historical fact, because I think it collapses three of the six questions from open decisions into cleanup. The provider-request capture had a production reader. I deleted the reader in #2605 and left the producer running. The broken link
That consumer was #1268 was also explicit that it did not want a second snapshot:
#615 had not landed, so #1277 wrote the capture as a stand-in for it. Then #2605 (merged 2026-08-11) deleted the reader and kept the producer: #2605's own summary said it would delete "legacy readers, duplicate runtime models, and their reverse dependencies". The capture producer was the reverse dependency it missed. That is my omission, not a retention policy anyone chose. The same shape repeated seven weeks later. #4300 (merged 2026-08-31) rebuilt the per-segment detail for Workstream 2 of #4299; #4302, the only consumer that needed it, was closed 2026-09-02. Two producers are still running. Neither has a reader. On scope, so this is not read as one story: there are two independent chains here, and I own one of them. The metadata amplification #4037 describes came from a different line — whole-record rewrite was the correct shape for the None of these changes was wrong when it landed, and that is the part worth naming. Each one is defensible inside its own PR; the cost only becomes visible across them. Several of us built these paths and share this, which is exactly why an audit that looks at the whole chain at once is worth having. Independent reproductionSame-schema synthetic benchmark — current Consistent with the 256× reported above. Measurements below are from one long-lived local installation of mine — an 814 MB workspace, not the 77 GB machine: Deduplicating every message by content within its own session, across all 379 capture files: Step N's prompt is step N−1's plus a few new messages. Storing each step in full re-serializes the same conversation hundreds of times, so the artifact is itself a write-amplification mechanism. The capture is also the fuel for #4037379 of my 436 artifacts (87%) are provider captures, and metadata WAL accumulates with the square of the artifact count. Simulating growth from zero using my real row shape: So the capture is not only ~95% of resident bytes; removing it also removes most of the amplification #4037 describes. These are not independent problems, and the ordering matters for how much #4037 buys on its own. What I think this changesQ3 — should #4037 land immediately: yes, unconditionally — and half of it now has. It is a defect independent of any retention policy. #4716 has since replaced Q1 — is exact provider-request reconstruction a guarantee: I do not think this is a retention decision yet. The current artifact should first be removed as unfinished cleanup from #2605. It has no reader, and it never satisfied #615 either: #615's fix plan item 2 asks for a content-addressed capture of the resolved system prompt, turn tail, effective Once it is gone, the retention question becomes a clean one about a capability we would be adding, with a purpose, quota and retention period stated up front — rather than a decision about whether to keep something already on disk. Q2 — is Q5 — the branch/copy/export matrix is smaller than it looks, but by one row, not two. Once Correcting myself on the second one: I had also expected the I have no disagreement on Q4 and Q6, and I think Q6 is the cheapest thing on this list to land first — a byte budget per physical model call, independent of prompt length, history length and artifact population, is what keeps any of this from growing back. The mechanism, in one pictureflowchart LR
REQ(["one provider request"])
REQ --> CAP["full request body<br/>private Artifact<br/>2.04 MB per step"]
REQ --> OBS["per-segment detail<br/>model-call event<br/>28.2 KB per attempt"]
OBS --> USE["Usage projection<br/>copies the whole attempt<br/>27.9 KB per row"]
CAP --> META["Artifact metadata<br/>full-table rewrite<br/>cost grows with total artifacts"]
CAP -.- D1["reader deleted<br/>PR 2605 · 2026-08-11"]
OBS -.- D2["reader closed<br/>PR 4302 · 2026-09-02"]
USE -.- D3["field never queried<br/>by Usage"]
classDef dead fill:#fcebeb,stroke:#e24b4a,color:#a32d2d
classDef live fill:#e1f5ee,stroke:#1d9e75,color:#0f6e56
classDef cost fill:#faeeda,stroke:#ba7517,color:#854f0b
class D1,D2,D3 dead
class CAP,OBS,USE live
class META cost
The green boxes are all written on every physical model call. None of them has a production reader for the part that dominates its bytes. The amber box is what turns the first of them into an O(N) cost that grows with everything ever written. What I have put up#4722 removes the capture producer, reclaims the captures already on disk, and folds the per-segment detail onto the record. Measured against its parent on the same machine:
The curve matters more than any single row: before, 4x the turns costs 8x the bytes, because each call copies a conversation that is itself growing. After, 4x the turns costs exactly 4x. Reclaiming the residue is in the same PR rather than left as follow-up, because captures are AI assistance disclosure: Claude Code helped trace the repository history, run the synthetic SQLite comparison and the local measurements, and draft this reply. I reviewed the evidence and own the conclusions. 简体中文审计我独立复现过一遍，结论成立。我想补一个历史事实，因为它能把六个问题里的三个从"待决策"降级成"收尾工作"。 provider 请求快照原本是有生产消费者的。消费者是我在 #2605 里删掉的，生产者留了下来。 断掉的那条链
那个消费者是 #1268 也明确说过不要第二份快照：
当时 #615 还没实现，所以 #1277 顺手把 capture 写了，作为它的占位。 然后 #2605（2026-08-11 合并）删掉了消费者，留下了生产者： #2605 自己的描述里写着要删除 "legacy readers, duplicate runtime models, and their reverse dependencies"。capture 的生产者正是那个被漏掉的 reverse dependency。这是我的遗漏，不是谁做过的保留策略决定。 七周后同样的形状又出现了一次。#4300（2026-08-31 合并）为 #4299 的 Workstream 2 重建了逐段明细；唯一需要它的消费者 #4302 在 2026-09-02 被关闭。两个生产者都还在跑，都没有读者。 关于范围，免得被读成一件事：这里其实是两条互不相干的链，我只占其中一条。#4037 描述的 metadata 放大来自另一条线 —— 整份重写对它最初面向的 这些改动落地时没有一个是错的，我觉得这才是值得点出来的地方。每一个放在自己的 PR 里都站得住，代价只有横跨着看才显形。这几条路径是我们几个人一起搭起来的，责任也是共同的 —— 这恰恰说明，一次把整条链摆在一起看的审计是有价值的。 独立复现同 schema 的合成对比 —— 当前 与上面报告的 256 倍处在同一量级。 以下测量来自我自己一个长期使用的本地环境 —— 814 MB 的 workspace，不是那台 77 GB 的机器： 把全部 379 个 capture 文件里的每条消息，按内容在各自 session 内去重： 第 N 步的 prompt 就是第 N−1 步加几条新消息。每步存全量，等于把同一段对话重新序列化了几百遍 —— 这份 artifact 本身就是一个写放大机制。 capture 同时也是 #4037 的燃料我这 436 个 artifact 里有 379 个（87%）是 provider capture，而 metadata WAL 随 artifact 数量的平方累积。用我这边真实的行形状模拟从 0 增长： 所以 capture 不只是约 95% 的常驻字节；把它去掉，同时也去掉了 #4037 描述的大部分放大。这两件事不是独立问题，先后顺序会明显影响 #4037 单独能拿到多少收益。 我认为这会改变什么Q3 —— #4037 是否应该立即处理：是，无条件 —— 而且现在已经做掉一半。 它是一个与任何保留策略都无关的缺陷。 #4716 已经把 Q1 —— 精确还原 provider 请求是否是基本保证：我认为现在还不到做保留决策的时候。 当前这份 artifact 应该先作为 #2605 的收尾被移除。它没有读者，也从未满足过 #615：#615 fix plan 第 2 条要的是内容寻址地捕获 resolved system prompt、turn tail、生效的 把它移除之后，保留问题就变成一个干净的问题：我们要不要新增这样一项能力，以及它的用途、配额和保留期限是什么 —— 而不是"要不要留住已经在磁盘上的东西"。 Q2 —— Q5 —— branch / copy / export 的矩阵确实比看上去小，但只小一行，不是两行。 第二项我要更正自己：我原本也以为 Q4 和 Q6 我没有异议。我认为 Q6 是这份清单里最便宜、最该先落地的一项 —— 为一次物理模型调用定一个与 prompt 长度、历史长度和 artifact 总数都无关的字节预算，是让上面这些东西不再长回来的唯一办法。 一张图说清机制flowchart LR
REQ(["一次 provider 请求"])
REQ --> CAP["完整请求正文<br/>私有 Artifact<br/>每步 2.04 MB"]
REQ --> OBS["逐段明细<br/>model-call 事件<br/>每条 28.2 KB"]
OBS --> USE["Usage 投影<br/>整个 attempt 再存一遍<br/>每行 27.9 KB"]
CAP --> META["Artifact metadata<br/>全表重写<br/>代价随 artifact 总数增长"]
CAP -.- D1["读者已删除<br/>PR 2605 · 2026-08-11"]
OBS -.- D2["读者已关闭<br/>PR 4302 · 2026-09-02"]
USE -.- D3["Usage 查询<br/>从不使用该字段"]
classDef dead fill:#fcebeb,stroke:#e24b4a,color:#a32d2d
classDef live fill:#e1f5ee,stroke:#1d9e75,color:#0f6e56
classDef cost fill:#faeeda,stroke:#ba7517,color:#854f0b
class D1,D2,D3 dead
class CAP,OBS,USE live
class META cost
绿色的三个框在每次物理模型调用时都会写。占据它们绝大部分字节的那一部分，都没有生产读者。琥珀色的那个框，是把其中第一个变成随历史无限增长的 O(N) 代价的原因。 我已经提交的#4722 移除 capture 生产者、回收磁盘上已有的 capture，并把逐段明细折叠到记录上。同机对照它的父提交实测：
比任何单行数字更重要的是曲线：before 是 4 倍轮数对应 8 倍字节，因为每次调用复制的那份对话本身在增长；after 是 4 倍轮数正好 4 倍。 存量回收放在同一个 PR 里而不是留作后续，是因为 capture 是 AI 使用说明：Claude Code 协助追溯仓库历史、完成 SQLite 合成对比与本地测量，并起草了这份回复。我复核了证据，并对这里的结论负责。 |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
English
Why discuss this now
This is a discussion about retention semantics and measurable budgets, not a proposal to adopt a particular database, object store, or migration plan.
A long-lived local installation recently reported:
runtime.sqlite: 1.5 GB, of whichcore_agent_run_eventsis about 1 GB;model_call_attempt_recordedevent: about 47 KB;artifacts/: 8.1 GB across 14,361 files, with a provider-request snapshot of about 900 KB written on each step;I audited current
main@b39e8d3. The observed numbers line up with several independent representations of the same call plus one scale-dependent metadata write path:What current
mainactually writesA complete request snapshot is still produced by default. Runtime Host always supplies
persistPreparedRequestArtifact, and each distinct provider step starts aprovider_request_captureArtifact write (composition, wiring, capture path). I found no production content reader for this Artifact. Its only production references are creation, codecs, optional joins, and conversation-copy rewriting. It has no TTL or automatic retention ceiling.The supposedly bounded observation explains the ~47 KB event rows.
PreparedRequestObservationretains up to 256 entries, each carrying kind, index, cacheability, comparison mode, SHA-256 digest, byte count, and optional labels (contract, construction). A schema-shaped 256-entry fixture serializes to 46,732 bytes before the surrounding attempt/event envelope. The only current production consumer I found immediately folds those entries into four kind totals plus a bounded tool breakdown (fold). The proposed detailed continuity consumer was closed in #4302 because its product value had not been established.Usage stores the entire attempt a second time. The Usage table is explicitly a rebuildable projection, but it serializes the complete
ModelCallAttemptintousage_model_call_attempts.record_json(authority contract, write). Usage readers do not userequestObservationorcaptureArtifactId, so the large diagnostic portion is persisted twice without serving a second query.Every Artifact mutation rewrites the complete metadata table.
ArtifactStorebuilds a new in-memory array and callsreplaceAll; the SQLite repository then runsDELETE FROM artifact_recordsand re-inserts every row in one transaction (caller, repository). This means a new 900 KB capture also causes O(all Artifacts) metadata WAL traffic.Copies amplify the retained history. Conversation copy selects every Artifact belonging to the copied turns and physically copies each payload (copy path); Session Bundle export includes every Session Artifact (bundle path). Canonical model-call events are also cloned, including their request observations.
There is another automatic Artifact write whose current product role needs confirmation. After a
Write,Edit, orBash > filecall settles, Runtime derives asource: 'tool_result'file or diff copy (producer, call site). This is not the context-pruning archive path. Pruning createssource: 'tool_result_archive', places amaka://archive/...reference andArchiveReadinstructions in the model-visible placeholder, and its reader rejects every other source (archive writer, source check). A caller that already knows an id can readtool_resultthrough the generic Artifact query API, so saying that it has no production reader would be too strong. However, Desktop hides this source by default and this automatic producer does not place anArchiveReadreference into ordinary model context. We should identify the product flow that still depends on these derived copies before deciding their retention. Every such write still triggers the full metadata rewrite above.Streaming partials are not the first target: current Runtime already coalesces them into bounded segments and removes the partial snapshot after the final event. The reported capacity is instead concentrated in provider captures and model-call events.
Does this explain the 77 GB?
I ran a controlled synthetic test using the current
artifact_recordsschema and indexes, WAL mode,synchronous=FULL, and 14,361 representative metadata rows:replaceAllfor one mutation: 9,484,272 WAL bytes, about 84 ms;If the full-rewrite cost grew roughly linearly from zero to that final size, 14,361 creates would account for about 68.1 GB of metadata WAL. Adding the observed 8.1 GB payload population gives about 76.2 GB, close to the reported 77 GB cumulative writes.
This is an inference from a same-schema synthetic benchmark, not I/O attribution measured on the affected machine. It is nevertheless strong evidence that #4037 is a present write-amplification problem, not merely a fallback optimization for a future Artifact retirement.
The unresolved semantic choice
Two existing goals appear to pull in different directions:
Both may be valid, but perhaps not under one default retention class. The current middle state pays the permanent storage cost while still not being replay-grade: the Artifact write is optional and best-effort, may finish without a durable join, represents the secret-free AI SDK request rather than the provider's exact wire request, and has no production content reader.
The decision I think is missing is therefore:
Possible invariants to debate
These are candidate decision criteria, not a settled implementation:
Questions
{digest, byte count, folded composition}sufficient? Is there a current consumer that needs every per-segment digest in every historical attempt?Relationship to existing work
This does not replace the existing threads:
The narrower question here is the cross-cutting retention contract and budget that those implementations should satisfy.
AI assistance disclosure: OpenAI Codex helped audit the current write paths, run the synthetic SQLite comparison, and draft this discussion. I reviewed the evidence and own the conclusions and questions.
简体中文
先把问题说清楚
这次先不谈该换 SQLite、对象存储还是别的后端。真正需要定下来的是两件事：一次普通运行究竟值得永久留下哪些数据，以及每次调用允许花多少磁盘成本。
最近有个长期使用的本地环境已经把问题暴露得很具体：
runtime.sqlite1.5 GB，core_agent_run_events自己占了约 1 GB；model_call_attempt_recorded大约 47 KB；artifacts/有 14,361 个文件，共 8.1 GB，几乎每一步都会保存一份约 900 KB 的 provider 请求；我按当前
main@b39e8d3把整条写入链路顺了一遍。结论很直接：同一份请求被换着形态保存了好几次；与此同时，每新增一个 Artifact，metadata 又会把已有记录从头抄一遍。目前可以确认的事实
第一，完整请求正文仍然默认落盘。 Runtime Host 无条件接入
persistPreparedRequestArtifact，每个不同的 provider step 都会异步写一份provider_request_capture（组装、接线、捕获路径）。我没有找到任何会读取其正文的生产代码；它现在只参与创建、编解码、可选关联和复制时的 ID 改写，也没有自动清理或容量上限。第二，47 KB 的 event 不是偶然。
PreparedRequestObservation最多保存 256 段，每段都带类型、序号、缓存属性、比较方式、SHA-256、字节数和可选标签（数据结构、生成过程）。按这个结构造满 256 段，光 observation 就有 46,732 字节，外层 attempt 和 event 还没算。可现在真正用到它的地方，只是马上把明细汇总成四类总量和一份有上限的工具排行（汇总逻辑）。原本可能使用逐段 digest 的连续性诊断 #4302 已经关闭，原因正是还没有证明这项能力值得长期维护。第三，Usage 把这块大字段又照单全收了一遍。 Usage 明明是可重建的查询投影，写入时却仍把完整
ModelCallAttempt放进usage_model_call_attempts.record_json（权威关系、写入代码）。Usage 查询根本不用requestObservation和captureArtifactId。也就是说，这几十 KB 不是为了支持另一种查询，只是因为 projection 复制了整个上游对象。第四，Artifact metadata 的写法会把任何小写入放大。 当前流程是先拼出完整记录数组，再调用
replaceAll；SQLite 随后删掉artifact_records全表，并把所有记录逐条插回去（调用端、repository）。所以新增一份 900 KB 请求文件，代价不只有这 900 KB，还包括一次随 Artifact 总数增长的 metadata WAL。第五，复制和导出没有把诊断数据排除在外。 Conversation copy 会把相关 turn 的 Artifact 逐个复制（复制路径），Session Bundle 也会把该 Session 的 Artifact 全部带上（导出路径）。model-call event 自然也会跟着复制，其中包含同一份 request observation。
第六，这里需要把两条名字相近、用途不同的路径分开。
Write、Edit、Bash > file调用结束后，Runtime 会自动派生一份source: 'tool_result'的文件或 diff（生产者、调用位置）。这和上下文裁剪无关。模型在裁剪后可以主动读取的，是另一种source: 'tool_result_archive'：Runtime 会在占位符里放入maka://archive/...引用和ArchiveRead提示，读取端也会拒绝其他 source（归档写入、source 校验）。如果调用方已经知道 ID，Host 的通用 Artifact 查询接口确实能够读取tool_result，所以“没有生产 reader”这个说法太满了。不过 Desktop 默认不会展示它，自动派生时也不会把ArchiveRead引用放进普通模型上下文。是否继续保存这些副本，应先确认现在还有哪个产品流程依赖它们；无论最终是否保留，每次写入仍会触发前面的 metadata 全表重写。Streaming partial 已经按有上限的 segment 合并，final event 写完后临时快照也会删除。它不是这次最该先动的地方。
77 GB 大概率从哪里来
我照当前
artifact_records的 schema 和索引做了一个合成测试，同样使用 WAL 与synchronous=FULL，表里放 14,361 条接近真实形状的 metadata：replaceAll新增一次记录，WAL 增加 9,484,272 字节，耗时约 84 ms；如果把记录从 0 增长到 14,361 的过程按近似线性估算，单是 metadata WAL 就会累计到 68.1 GB 左右；再加已经看到的 8.1 GB payload，合计约 76.2 GB，和进程统计的 77 GB 几乎处在同一个位置。
这里要说严谨一点：这是同 schema 合成测试支持下的强推断，并不是对那台机器做了逐笔 I/O 归因。但它足以说明 #4037 不是一个可有可无的后备优化，而是现在就在发生的主要写放大。
要讨论的其实不是压缩算法
现有两个目标各自都有道理：
问题在于，当前状态两头都没占到：普通运行承担了永久保存完整请求的成本；真要拿它做严格证据，它又是可选、best-effort 的，可能文件已经写成但 attempt 没关联上，而且保存的是脱敏后的 AI SDK 请求，不是 provider 最终收到的 wire request。更现实的一点是，现在也没有生产代码读取这些正文。
所以我觉得应该先把这个问题单独说透：
希望这次能先形成的共识
下面都只是讨论标准，不是实施结论：
想请大家重点回答这几个问题：
{digest、字节数、汇总后的 composition}是否已经够用？现在是否真有功能需要永久保存每次 attempt 的逐段 digest？和已有工作的边界
我查过现有讨论和 issue，没有找到一个已经覆盖上述完整问题的线程：
这里不替代它们，也不提前决定它们怎么实现。这里缺的是一把共同的尺子：正常运行到底该留下多少东西、什么数据值得长期保存、后面的实现要守住怎样的容量上限。
AI 使用说明：OpenAI Codex 协助核对当前写入路径、完成 SQLite 合成对比，并起草了这份讨论。我复核了证据，并对这里的判断和问题负责。
All reactions