Skip to content

QuiverDB v1.2.0

Choose a tag to compare

View all tags
@artemonad artemonad released this 05 Oct 23:05
· 3 commits to master since this release
v1.2.0
68c2016

QuiverDB v1.2.0 — Snapshot Isolation (Phase 1) + Backup/Restore

On‑disk formats remain frozen (no migrations):

  • Meta v3, Page v2, WAL v1

This release delivers in‑process snapshot isolation (as‑of LSN) and backup/restore built on top of snapshots, plus a micro‑fix to WAL replay to reduce unnecessary I/O.

Highlights

  • Snapshot Isolation (Phase 1)
    • Db::snapshot_begin() -> SnapshotHandle { lsn }
    • SnapshotHandle::get/scan_all/scan_prefix — consistent reads “as of LSN”
    • Page‑level COW: before overwriting/freeing a v2 page needed by a live snapshot, the current image is frozen to a sidecar
    • Readers pick live vs frozen per page; overflow chains resolved per page the same way
    • Rare fallback scan reconstructs tail‑wins when chain heads move after the snapshot
  • Backup/Restore
    • Full backup: all pages as‑of snapshot_lsn
    • Incremental backup: pages with page_lsn in (since_lsn, snapshot_lsn]
    • Archive layout:
      • pages.bin — frames [page_id u64][page_lsn u64][page_len u32][crc32 u32] + payload(page_size)
      • dir.bin — directory copy (if present)
      • manifest.json — summary (snapshot_lsn, since_lsn, page_size, counters)
    • Restore writes pages back (raw), installs dir.bin if present, sets last_lsn and clean_shutdown
  • Micro‑fix: WAL replay optimization
    • Startup replay now performs LSN‑gating before ensure_allocated for v2 pages
    • Matches wal‑apply behavior and avoids unnecessary segment allocations on stale frames
  • Metrics for snapshots/backup/restore
    • snapshots_active, snapshot_freeze_frames/bytes
    • backup_pages_emitted/bytes, restore_pages_written/bytes

CLI

  • Backup/Restore (Phase 1)
    • quiverdb backup --path ./db --out ./backup [--since-lsn N]
    • quiverdb restore --path ./dst --from ./backup [--verify]
  • CDC (unchanged wire format; reminder)
    • wal-ship (stdout/TCP/file, since-lsn, optional gzip/zstd)
    • wal-apply (idempotent, LSN‑gating, partial‑tail tolerant)
    • cdc record/replay (deterministic slices), cdc last-lsn

Examples

  • Full backup and restore
    • quiverdb backup --path ./db --out ./backup
    • quiverdb restore --path ./db_restored --from ./backup --verify
  • Incremental backup since LSN
    • quiverdb backup --path ./db --out ./backup_incr --since-lsn 12345

Rust API (snapshot)

let mut db = Db::open(root)?;
db.put(b"k", b"v0")?;
let mut s = db.snapshot_begin()?;
db.put(b"k", b"v1")?;
assert_eq!(s.get(b"k")?.as_deref(), Some(b"v0".as_ref()));
db.snapshot_end(&mut s)?;

Compatibility

  • No on‑disk format changes (meta v3, page v2, WAL v1)
  • CDC wire format unchanged; older appliers remain forward‑compatible (unknown types ignored, partial tails tolerated)

Known limitations (Phase 1)

  • Snapshots are in‑process (not persisted across restarts)
  • No backup deduplication yet (planned in a later phase)
  • Freeze sidecar prioritizes latency (CRC‑checked on read; not fsync’d per frame by design)

Upgrade notes

  • No migration required; rebuild and run
  • If you ship compressed CDC streams, ensure wal-apply uses matching decompression (gzip/zstd)

Docs

  • See docs/snapshots.md for semantics, COW details, sidecar formats, and backup/restore
  • See docs/api.md and docs/cdc.md for updated guides

Thank you for trying QuiverDB!

Assets 2
Loading