Skip to content

QuiverDB v1.1.0

Choose a tag to compare

View all tags
@artemonad artemonad released this 05 Oct 02:24
· 3 commits to master since this release
v1.1.0
f84dac4

Summary

  • Formats unchanged and frozen: meta v3, page v2, WAL v1.
  • Focus: easier integrations (CDC), better ergonomics (builder, scans), observability, and tooling.
  • 100% backward compatible with v1.0.0 databases (no migration).

Highlights

  • Config/Builder: QuiverConfig and DbBuilder with explicit open_with_config/open_ro_with_config.
  • Live subscriptions (in-process) for put/del events with LSN.
  • Scans: scan_all/scan_prefix and CLI dbscan with optional JSON.
  • CDC upgrades: wal-ship tcp:// sink and --since-lsn, deterministic cdc record/replay, cdc last-lsn helper.
  • Metrics: metrics --json and metrics-reset.

What’s new
Added

  • Configuration and builder
    • QuiverConfig + DbBuilder with explicit Db::open_with_config and Db::open_ro_with_config.
    • Tunables: wal_coalesce_ms, data_fsync, page_cache_pages, ovf_threshold_bytes.
    • Builder takes precedence over environment variables (env still supported).
  • In‑process subscriptions
    • Db::subscribe_prefix(prefix, callback) → SubscriptionHandle (Drop = unsubscribe).
    • Event { key, value: Option<Vec>, lsn } — value=None for delete.
    • Callbacks run synchronously after a successful commit (keep handlers fast).
  • Scans
    • Db::scan_all() and Db::scan_prefix(prefix) with tail‑wins semantics.
    • CLI: dbscan [--prefix X] [--json].
  • CDC tooling
    • wal-ship: --since-lsn N (exclusive) and --sink=tcp://host:port (built-in TCP sink).
    • cdc last-lsn: print follower’s last LSN for resume.
    • cdc record: save deterministic wire-format slices with optional --from-lsn/--to-lsn.
    • cdc replay: apply slices (or stdin) with optional LSN filters.
  • Metrics CLI
    • metrics --json (one-line snapshot) and metrics-reset.

Changed

  • WAL writer group-commit: shared registry coalesces fsync across appenders in the same process.
  • dbstats now prints a metrics snapshot alongside DB stats.

Hardening and reliability

  • CDC/wal-apply: tolerant to partial tails, ignores unknown record types (forward-compatible).
  • LSN-gating on v2 pages prevents older frames from overwriting newer data.
  • Directory: updates remain atomic (tmp+rename); header CRC verified on open.
  • Repair: conservative freeing of orphan overflow chains.

Compatibility

  • No on-disk format changes; 1.0.0 databases work as-is.
  • Ship wire format preserved; unknown record types are safely ignored by consumers.
  • wal-apply is idempotent (safe to re-apply the same frames).

Upgrade notes

  • If you use only env vars, behavior is unchanged.
  • Prefer the new builder in application code:
    • Use DbBuilder and Db::open_with_config/Db::open_ro_with_config to override env in code.
  • Subscription callbacks run synchronously in the writer thread; keep them lean or hand work off to another thread/task.

Quick start snippets

  • Builder and explicit open:

    use QuiverDB::{Db, Directory, init_db};
use QuiverDB::config::DbBuilder;

let root = std::path::Path::new("./db");
if !root.exists() {
    init_db(root, 4096)?;
    Directory::create(root, 128)?;
}
let cfg = DbBuilder::new()
    .wal_coalesce_ms(0)
    .data_fsync(true)
    .page_cache_pages(256)
    .build();

let mut db = Db::open_with_config(root, cfg.clone())?;
db.put(b"alpha", b"1")?;
let db_ro = Db::open_ro_with_config(root, cfg)?;
assert_eq!(db_ro.get(b"alpha")?.unwrap(), b"1");

  • CDC TCP + resume:

    # Follower checkpoint
LSN=$(quiverdb cdc last-lsn --path ./follower)
# Leader: ship only frames with lsn > $LSN to a TCP sink
quiverdb wal-ship --path ./leader --since-lsn $LSN --sink=tcp://127.0.0.1:9999 --follow
# Follower: apply
nc -l -p 9999 | quiverdb wal-apply --path ./follower

  • Deterministic record/replay:

    quiverdb cdc record --path ./leader --out ./slice.bin --from-lsn 100 --to-lsn 200
quiverdb cdc replay --path ./follower --input ./slice.bin --from-lsn 100 --to-lsn 200

  • dbscan (JSON):

    quiverdb dbscan --path ./db --json
quiverdb dbscan --path ./db --prefix k --json

Docs

  • API: config/builder, scans, subscriptions, CLI mapping — docs/api.md (v1.1.0).
  • CDC: TCP sink, since-lsn, record/replay, last-lsn — docs/cdc.md.
  • Format: unchanged — docs/format.md.

Checks

  • cargo test --all --release
  • On Windows, directory fsync after rename is best‑effort; covered by tests.
Assets 2
Loading