Skip to content

feat(snapshots): add lossless v5 compression#31

Merged
colinvkim merged 4 commits into
mainfrom
feat/snapshot-v5-compression
Jul 25, 2026
Merged

feat(snapshots): add lossless v5 compression#31
colinvkim merged 4 commits into
mainfrom
feat/snapshot-v5-compression

Conversation

@colinvkim

@colinvkim colinvkim commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Summary

  • add snapshot format v5 with bounded streaming LZFSE compression for node and topology sections
  • preserve exact snapshot semantics, the .radixscan package experience, cheap manifest and stats previews, and v3/v4 import compatibility
  • verify decoded SHA-256 checksums for every v5 body section plus stored byte counts to reject corruption, truncation, and trailing transport data
  • keep v4 and v5 snapshots in the same normalized comparison pipeline with zero cross-version deltas for equivalent scans
  • document the wire contract, rollback strategy, rejected deterministic-key experiment, and release evidence in SNAPSHOT_V5_PLAN.md

Validation

  • 56 focused archive tests pass, including v3/v4/v5 round trips, cross-version comparisons, Unicode and identity fields, malformed streams, integrity metadata, decoded-size bounds, cancellation, preview behavior, and atomic replacement
  • complete Swift suite passes: 717 tests, 18 skipped, 0 failures; the final suite passed three consecutive stress runs
  • complete macOS Debug app build passes
  • localization catalog passes for en, de, es, fr, it, and zh-Hans

Benchmark results

Same-process v4/v5 medians across three debug iterations:

Shape Nodes v5 / v4 size Export ratio Import ratio
Wide 10,001 6.8% 1.05x 0.67x
Deep 12,001 4.8% 1.06x 0.66x
Large 64,065 8.6% 1.08x 0.78x

The 7-node fixture grows by 223 bytes because the richer v5 manifest outweighs its compressed body savings. No tiny-archive shrinkage claim is made. The implementation adds no dependency and uses the macOS Compression framework.

Summary by CodeRabbit

  • New Features
    • Added Snapshot Format v5 with lossless, losslessly compressed (LZFSE) nodes/topology sections.
    • Added per-section encoding metadata, plus per-section integrity checksums and stored byte counts for v5.
    • Enabled configurable export format selection for choosing v5 vs earlier formats.
  • Documentation
    • Updated docs to describe v5 streaming/section compression, reopening exact snapshots, and cross-version compatibility.
    • Added a v5 plan covering validation, corruption handling, and expected losslessness behavior.
  • Bug Fixes
    • Strengthened detection and localized reporting for corrupted, truncated, incomplete, or malformed archives (including streaming/compression failures).

Copilot AI review requested due to automatic review settings July 24, 2026 01:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 465ad010-60de-475b-90a3-bae72cdac788

📥 Commits

Reviewing files that changed from the base of the PR and between c293140 and aacca0e.

📒 Files selected for processing (6)
  • Radix/Services/ScanArchiveModels.swift
  • Radix/Services/ScanArchiveNodeIO.swift
  • Radix/Services/ScanArchiveService.swift
  • RadixCoreTests/AppModelDependencyTests.swift
  • RadixCoreTests/ScanArchiveServiceTests.swift
  • SNAPSHOT_V5_PLAN.md
🚧 Files skipped from review as they are similar to previous changes (5)
  • Radix/Services/ScanArchiveModels.swift
  • SNAPSHOT_V5_PLAN.md
  • Radix/Services/ScanArchiveService.swift
  • RadixCoreTests/ScanArchiveServiceTests.swift
  • Radix/Services/ScanArchiveNodeIO.swift

📝 Walkthrough

Walkthrough

Format v5 adds LZFSE-compressed archive sections, decoded-byte checksums, stored byte counts, and version-aware import/export. Shared streaming IO, manifest models, validation, localization, documentation, tests, and benchmarks are updated accordingly.

Changes

Snapshot Format v5

Layer / File(s) Summary
Archive contracts and section streaming
Package.swift, Radix/Services/ScanArchiveModels.swift, Radix/Services/ScanArchiveSectionStream.swift, Radix/Localizable.xcstrings
Adds v5 integrity metadata and shared identity/LZFSE section readers and writers with checksum tracking, cancellation, truncation, trailing-data, progress, and localized error handling.
Encoding-aware node and topology IO
Radix/Services/ScanArchiveNodeIO.swift
Routes node and topology serialization through encoding-aware section streams, returns finalized checksums, and adds cancellation checks during large traversals.
Versioned archive export and import
Radix/Services/ScanArchiveService.swift, README.md
Adds format version selection, compressed section filenames, v5 manifests, encoding resolution, byte-count validation, per-section checksum verification, and updated archive documentation.
Compatibility validation and release support
RadixCoreTests/*, SNAPSHOT_V5_PLAN.md
Adds v4/v5 round-trip, compression, corruption, integrity, preview, benchmark, compatibility, and losslessness coverage, plus the v5 design and validation plan.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

Suggested labels: enhancement

Sequence Diagram(s)

sequenceDiagram
  participant ScanArchiveService
  participant ScanArchiveSectionWriter
  participant ScanArchiveDocument
  participant ScanArchiveSectionReader
  participant Snapshot
  ScanArchiveService->>ScanArchiveSectionWriter: export encoded nodes and topology
  ScanArchiveSectionWriter-->>ScanArchiveService: return decoded-byte checksums
  ScanArchiveService->>ScanArchiveDocument: write v5 encodings, checksums, and byte counts
  ScanArchiveService->>ScanArchiveSectionReader: read and decode archive sections
  ScanArchiveSectionReader-->>ScanArchiveService: return decoded section bytes
  ScanArchiveService->>Snapshot: import validated snapshot
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 1.19% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding lossless snapshot format v5 compression.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/snapshot-v5-compression

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Radix/Services/ScanArchiveService.swift`:
- Around line 918-924: Update readSectionData’s catch chain to explicitly
rethrow CancellationError before the generic catch maps other failures through
mapError. Preserve the existing ScanArchiveError passthrough and domain-error
mapping for non-cancellation errors.

In `@SNAPSHOT_V5_PLAN.md`:
- Around line 496-502: Update the regression-suite commands in
SNAPSHOT_V5_PLAN.md to prefix both swift test and the multiline xcodebuild
invocation with rtk, matching the documented repository policy and the later
benchmark command.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d0ab57da-4128-40e8-b191-5d5abe545664

📥 Commits

Reviewing files that changed from the base of the PR and between 7b5e05a and c293140.

📒 Files selected for processing (10)
  • Package.swift
  • README.md
  • Radix/Localizable.xcstrings
  • Radix/Services/ScanArchiveModels.swift
  • Radix/Services/ScanArchiveNodeIO.swift
  • Radix/Services/ScanArchiveSectionStream.swift
  • Radix/Services/ScanArchiveService.swift
  • RadixCoreTests/ScanArchiveBenchmarkTests.swift
  • RadixCoreTests/ScanArchiveServiceTests.swift
  • SNAPSHOT_V5_PLAN.md

Comment thread Radix/Services/ScanArchiveService.swift
Comment thread SNAPSHOT_V5_PLAN.md Outdated
@colinvkim

Copy link
Copy Markdown
Owner Author

Ready to merge after QC, I think. Perf gains are demonstrated but need some larger scale testing + with Sonoma. Hopefully this weekend!

@colinvkim
colinvkim merged commit 80be8c1 into main Jul 25, 2026
1 check passed
@colinvkim
colinvkim deleted the feat/snapshot-v5-compression branch July 25, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants