Pigeonhole 0.2.0: the wide-column model

Pigeonhole 0.2.0 completes the Phase 2 wide-column model of an embedded, single-file, wide-column store in Rust (BigTable's data model with SQLite's deployment model). It adds counter families, large values and blob separation, per-family compaction styles and zstd, and a file at rest near its live size. It also includes the read- and write-path work that met the Phase 2 performance gate as amended by D193.

Experimental 0.x. The on-disk format and the API may change before 1.0 (FORMAT.md). The latency work (Phase 3) is in progress, and Pigeonhole is not recommended for production use yet. See docs/status.md for the roadmap.

Upgrading from 0.1.0:

• Counters now live in counter families. incr on a Family::default() family fails with InvalidArgument; declare Family::counter(). Families in files written by 0.1.0 keep their 0.1.0 behavior.
• New file format. Files written by 0.2.0 use format version 2, which 0.1.0 can't open. Keep a backup if you may need to downgrade.
• Struct literals no longer compile for FamilyOptions and the model types (#[non_exhaustive]); use the setters.
• FUSE mounts are refused unless you opt in with allow_fuse(true).

The changelog has the steps.

What's new

• Counter families (D179, D186, D187), after Bigtable's aggregate families.
  • incr keeps a counter as one cell however often it is incremented.
  • incr_at and put_i64_at keep dated buckets, for hourly or daily totals.
  • Increments of one counter in one mutation or batch add up in order.
  • A delete restarts the counter from 0, and a bottommost compaction purges it.
• Large values. Values up to 4 GiB − 2 bytes (D188): a put above the inline limit goes to a blob file at commit time.
• Blob separation and blob GC. Values above a family's blob_threshold (default 4 KiB) move to blob files when flushed or compacted (D180). Blob GC follows the references each SST records (D184). backup copies separated values, and shrink relocates blob extents.
• Per-family compaction styles: Compaction::Tiered for write-heavy families, and Compaction::FifoByTime, which drops whole expired files on a timer (D168).
• zstd block compression with a per-family level (D175).
• A file at rest near its live size. After compact() and shrink(), a file above 5 MiB is 1.05–1.16× its live data, where it was 2–4× (D183, D185, D190).
• Flush-time version GC. A flush drops overwritten versions beyond max_versions and cells hidden at every snapshot, so hot rows reach disk already trimmed (D191). A bottommost compaction's purge runs under a guard against backfilled writes (D192).
• Custom merge operators work. Operators registered with Options::merge_operator reach the engine. A handle opened over an unregistered operator is read-only, as documented.
• The FUSE opt-in. Options::allow_fuse(true) accepts trusted local FUSE mounts (ntfs-3g, gocryptfs). Network filesystems are always refused (D173).
• Options::write_stall_timeout, and #[non_exhaustive] option and result structs with builder setters, so later fields are not breaking.

Performance

The Phase 2 gate (D193) uses phdb-bench sparse-wide --scale full: 10 shards, 64 MiB memtables, 256 MiB cache, one client thread. In the official runs 5–6 on a quiet 10-core Apple M5, Pigeonhole beats hand-keyed RocksDB on every measure and matches or beats SQLite EAV on throughput, get and put p99, and p99.9.

Store	ops/s	p99 µs	p99.9 µs	get p99	put p99	row-read p99	scan p99
Pigeonhole	28.7K / 30.4K	389 / 395	528 / 541	98 / 11	17 / 17	455 / 459	489 / 496
SQLite EAV	27.1K / 29.8K	245 / 250	1,040 / 938	156 / 152	178 / 172	222 / 231	293 / 307
RocksDB	20.4K / 20.4K	709 / 791	1,012 / 1,196	161 / 155	9 / 9	840 / 963	881 / 1,016

• The one documented gap is the p99 of reads of very wide, heavily overwritten rows: row reads about 2× SQLite EAV, scans about 1.6× (#387, Phase 3).
• The cause: an overwrite adds a version, and these reads step over the superseded versions until compaction removes them. A compacted hot row costs about the same per cell as SQLite.
• What to do now: the guide's Wide, overwritten rows says how to keep it down: read only the columns you need, set max_versions, and run compact() before read-heavy phases.

How it was tested

• Simulation. Deterministic simulation (SimVfs) with fault injection, process crashes and power loss, checked against a reference model with a record-level recovery oracle. The model runs exercise counter families, blob separation, large values and flush-time GC.
• Seed sweeps of 1–300 of the engine model check, tablets and public model suites on the changes that touch reads, writes, flushes and compaction.
• Other checks: Miri on the unsafe crates, loom on the concurrent ones, and CI on Linux, macOS and Windows.
• Performance regression checks. Every read- and write-path change was measured by CI's instruction counts (Linux callgrind) over hot-row, read and write shapes.
• D193's floors keep what Phase 2 reached. CI fails a change that makes any shape worse than its per-change threshold or puts it above its absolute ceiling. Every later official gate run, including the one before each release, is compared with the committed Phase 2 baseline (crates/bench/baselines/phase2-gate).
• Gate runs. The official sparse-wide gate runs 5–6, against SQLite EAV and RocksDB, on a quiet machine.

Known limits (tracked)

• Wide-row read tail: reads of wide, heavily overwritten rows have a p99 about 2× SQLite EAV for row reads and about 1.6× for scans. #387, Phase 3.
• Shard scaling: write throughput does not yet scale with shard count. #154, Phase 3.
• Open latency: about 14 ms against the 5 ms goal. #158, Phase 3.
• Application-owned mode still runs the default I/O backend's 2–16 I/O threads. Phase 3.
• The async API (async feature) is Phase 3.
• pigeonhole-cli (phdb) is a placeholder until Phase 4.

Crates

pigeonhole (start here), plus pigeonhole-engine, -format, -io, -pager, -wal, -memtable, -cache, -runtime, -shm, -sst, -compaction, -sim, -arrow and pigeonhole-cli, all at 0.2.0.

Docs

• Changelog: CHANGELOG.md, with migration notes.
• User guide: docs/guide covers getting started, concepts, data modeling (now with counter families and wide rows), durability, scans and filters, and errors. The agent reference is for coding agents.
• Design: the spec and the decisions, D168–D193 for Phase 2.