Proposal: Milestone 1.5 - High-Performance Storage & Restoration

[kavix]

Hi team!

I'd like to propose Milestone 1.5: High-Performance Storage & Restoration to focus entirely on advanced caching, compression, and restoration speedups.

We've already implemented and opened PRs for two major performance optimizations for this milestone:

1. Adaptive ZSTD Compression (perf: Implement Adaptive Compression (zstd with file-type heuristic & multi-threading) #124 - PR perf: implement adaptive zstd compression and raw binary storage #167):
   • Integrated github.com/klauspost/compress/zstd with sync.Pool decoder recycling.
   • Implemented binary/media checks to store already-compressed files raw (.raw), saving redundant CPU cycles.
2. Zero-Copy OS Reflinks (perf: Implement Zero-Copy OS Reflinks (clonefile on macOS / FICLONE on Linux) for instant restore #123 - PR perf: implement zero-copy OS reflinks using Strategy pattern #168):
   • Implemented macOS APFS clonefile and Linux ioctl(FICLONE) strategies.
   • Restores raw files instantly in constant-time bypassing memory copies.

📊 Verified Benchmark Results (Apple M2 arm64)

• Reflink Cloning vs Standard File Copy (5MB file):
  • Standard file copy: 9.09 ms
  • Zero-copy reflink: 0.128 ms (128 microseconds) — 70.8x faster!
• Zstd vs Gzip Decompression (1MB text file):
  • Gzip decompression: 0.364 ms
  • Zstd decompression: 0.333 ms — 10% faster (Zstd decompression bandwidth runs at ~1.2 GB/s).

🛠️ Remaining Roadmap for Milestone 1.5:

To complete Milestone 1.5, we should implement the remaining performance backlog items:

• FastCDC (Block-Level Content-Defined Chunking) (perf: Implement Block-Level Content-Defined Chunking (FastCDC) for delta-deduplication #122): Deduplicate blocks in large files so that minor edits only consume ~64KB of disk space instead of storing duplicate full files.
• Memory-Mapped Files (mmap) (perf: Use Memory-Mapped Files (mmap) for Zero-Copy Hashing & Fast Scanning #125): Zero-copy file hashing directly from the kernel page cache.
• Multi-Threaded Directory Walker (perf: Multi-threaded directory walking with Async I/O prefetch queue #126): Concurrent walking and double-buffered pre-fetching to fully saturate NVMe/SSD IOPS.

Please share your thoughts on these speedups and the Milestone 1.5 roadmap!