Skip to content

fix(storage): preserve context images in offline snapshots - #4965

Merged
Astro-Han merged 1 commit into
mainfrom
fix/4071-context-backup-integrity
Sep 7, 2026
Merged

fix(storage): preserve context images in offline snapshots#4965
Astro-Han merged 1 commit into
mainfrom
fix/4071-context-backup-integrity

Conversation

@likun666661

Copy link
Copy Markdown
Member

Summary

Refs #4071. Second slice after #4963, following the revised plan.

Read images already persist as Session context refs, but operational backups and Session exports only included runtime.sqlite and artifacts. A successful snapshot could therefore restore history without its image bytes.

  • Include a standalone context SQLite snapshot and its referenced managed values. Verify blob sizes/hashes, database integrity, usage and typed context references before publishing backups/exports and again before publishing a restore.
  • Hold the exclusive Storage Root owner across context snapshots, outside the existing Artifact lock. An active Host causes an explicit offline-required refusal; a Session quiescence interface alone does not fence global context GC.
  • Filter Session exports to that Session's context refs and deduplicated payloads. Drop unreferenced blobs and maintenance queues from the destination and rebuild its usage counters. Vacuum only the private destination to remove other Sessions' deleted inline bytes.
  • Preserve Session/ref identities for restoration into a separate root. No source RuntimeEvents or source context database are rewritten.
  • Emit operational backup format v4 while retaining v3 decoding/restoration. Reject old snapshots that actually carry dangling context refs rather than silently presenting them as complete.
  • Normalize the filtered runtime database to a standalone file so validation does not add WAL sidecars to the bundle inventory.

Verification

  • Core and storage builds passed; storage typecheck passed.
  • Full storage suite: 1191 passed, 8 skipped, 0 failed.
  • Final focused backup/export/production-snapshot/Artifact-lock suites: 32 passed, 0 failed.
  • Negative control: bypassing context copying makes the restored-image regression fail on missing context references; reverted before final validation.
  • New coverage: empty-root image restore, shared-blob deduplication, single-Session isolation including inline free-page bytes, active-owner refusal, exclusive-owner release after failure, missing database/files, corrupt bytes despite a recomputed manifest, cross-Session projection refs, opaque JSON non-interpretation, symlinked value ancestors and v3 backup compatibility.
  • Root lint/format check, ASF headers and diff check passed.

Not run: full repository build/typecheck, Runtime Host/Desktop suites, Linux/Windows runs, cross-process snapshot race injection or multi-GiB latency/RSS benchmarks.

Rollout and Scope

Context-bearing roots require offline backup/export in this slice. Stop the Runtime Host and release its root owner first. Calls through a live Host/quiescence adapter are refused, not treated as a globally consistent snapshot. Artifact-only roots retain their existing locking behavior; context appearing during such a snapshot forces a retry.

The context DB is copied using SQLite backup, migrated/filtered only in the private destination, and validated before publication. The source is not vacuumed or migrated. Root identity markers are not exported.

This covers standalone backup restore and the existing isolated Session bundle state exporter. It does not introduce merge-import into an already-running workspace, ref-ID remapping into an existing Session, an online snapshot protocol, a new UI/CLI surface, or automatic recovery of already-missing image bytes. Whole-root backup still has its existing operational-state scope; this is not a claim to include every independent product database.

Backup files written as v4 require this reader; older binaries need not accept them. Existing supported v3 backups remain readable when their references are complete.

AI use

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Codex inspected storage authorities, implemented snapshot/validation changes and tests, ran local verification and prepared this PR. Commit includes Generated-by: Codex.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — context images survive snapshots, and unsafe online context exports are refused.
  • No

Include verified context refs and managed bytes in backups and Session exports under exclusive root ownership. Preserve v3 backup reads, version new backups as v4, filter context data for a single Session, and reject missing references before publishing a snapshot.

Generated-by: Codex
@github-actions github-actions Bot added the effort/L Under 1000 readable lines label Sep 7, 2026

@alva-bot01 alva-bot01 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Slack

@alva-bot01 alva-bot01 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved via Slack

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I reviewed d2dce277485ae933202c386dcbfca98c67161669 with four independent reviewers and found no remaining P0–P2 issue.

The change addresses the shipped Read-image backup gap at the existing storage boundary. Context-bearing snapshots hold the exclusive root owner outside the Artifact lock, copy and normalize a private database, preserve Session/ref identities, and validate typed reference closure plus payload hashes before publication. Single-Session export removes unrelated context refs/payloads and vacuums the private copy. Supported v3 backups remain readable; dangling historical image refs fail closed.

Independent focused validation passed 12 tests covering snapshot/export/restore, shared blobs, private inline bytes, owner refusal/release, missing or corrupt payloads, symlink ancestors and v3 format compatibility. Local storage compilation reported TypeScript errors while emitting the tested dist, so this is not a clean local full-build claim; exact-head CI passed. I did not independently repeat the PR author's negative control, cross-process race injection, large-data benchmarks or cross-platform runs.

One optional P3 consistency check is noted inline. The proposed absent-to-present race did not establish an incomplete target snapshot: runtime.sqlite is captured earlier, and missing references in that captured state are rejected before publication. Later source writes alone do not invalidate an earlier complete snapshot.

AI disclosure: Codex assisted with source review, independent reviewers, focused validation and this review.

中文摘要

四路复核后没有剩余 P0–P2。修复接在现有 storage 边界:context 快照受独占 root owner 保护;仅处理私有副本,保留引用身份,发布前核对引用、长度和哈希;单 Session 导出过滤其他 Session 数据并压实副本。独立聚焦测试 12 项通过,v3 格式兼容与缺失引用拒绝已覆盖。保留一条 P3 派生计费校验建议。本地编译有 TypeScript 错误但产出了所测 dist,不能宣称完整构建通过;当前 CI 通过,未独立复跑作者负对照、大数据和跨平台验证。

.prepare(`SELECT 1 FROM (
SELECT r.session_id, count(*) AS reference_count, sum(b.size_bytes) AS logical_bytes
FROM context_refs r JOIN context_blobs b USING(blob_id) GROUP BY r.session_id
EXCEPT SELECT session_id, reference_count, logical_bytes FROM context_session_usage

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 — optional exactness check for derived Session usage. The one-way EXCEPT detects missing or differing expected rows but accepts an extra usage row for a Session with no refs. An independent probe inserted such a row and validation still succeeded. Normal snapshot generation rebuilds this table, so I did not find a supported generated-backup path that produces it; this is defensive validation for altered input, not a merge blocker or a claim of a new security boundary. Consider checking the reverse difference as well so the validator enforces exact equality.

中文摘要

P3 可选加强:单向 EXCEPT 会放过无引用 Session 的额外 usage 行,注入探针已确认。正常快照会重建该表,没有发现正常生成路径会产生此状态,因此不阻塞合并。可增加反向差集检查,让派生账目校验成为严格相等。

@Astro-Han
Astro-Han merged commit cde2423 into main Sep 7, 2026
2 checks passed
@Astro-Han
Astro-Han deleted the fix/4071-context-backup-integrity branch September 7, 2026 08:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/L Under 1000 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants