0.2.1: WAL crash-recovery fix

A fix to WAL crash recovery. Upgrade recommended for every 0.2.0 user. The file format is unchanged, and files written by 0.2.0 open as before. Every published crate moves to 0.2.1 together.

Fixed

• Records the WAL had lost could come back after a later recovery (#508, D209).
  • When: a power loss (not a process crash) while a WAL segment was starting in a reused slot. The disk had to keep part of that write (the first records) and lose the rest (the segment's header). The database then had to reopen, write little or nothing more to that WAL stream, and recover again later.
  • What happened: recovery correctly ignored those records the first time. But the reopened stream reused the lost header's numbering in the same slot, so at the next recovery the old records read as new ones.
  • Effect: writes that a power loss should have discarded could reappear. Their sequence numbers could also be reused by newer commits, mixing old and new data in recovery. The records that come back are only ever ones that were never synced.
  • The fix: recovery now starts a reopened stream three numbers above the largest it saw (FORMAT §10.1). Nothing reads or writes differently otherwise.

Upgrading

Change nothing but the version: pigeonhole = "0.2" already resolves to 0.2.1 after a cargo update -p pigeonhole. Process crashes were never affected on 0.2.0; the exposure is a power loss during a specific WAL write. Acknowledged durable commits (GroupSync / Sync) were never lost.

Details: #508 (root cause and reproduction), D209 (the fix), #512 (main), #514 (this backport).

Phase 3 gate results, 2026-10-11 (reference machine)

Full results tarball of the first Phase 3 gate run on the AX102 reference machine (#405, #406): every step's JSON, tables, raw logs and criterion reports. The readable results are committed under crates/bench/baselines/phase3-gate/results/2026-10-11/ (start with summary.md). This is a data archive, not a software release.

Pigeonhole 0.2.0: the wide-column model

Pigeonhole 0.2.0 completes the Phase 2 wide-column model of an embedded, single-file, wide-column store in Rust (BigTable's data model with SQLite's deployment model). It adds counter families, large values and blob separation, per-family compaction styles and zstd, and a file at rest near its live size. It also includes the read- and write-path work that met the Phase 2 performance gate as amended by D193.

Experimental 0.x. The on-disk format and the API may change before 1.0 (FORMAT.md). The latency work (Phase 3) is in progress, and Pigeonhole is not recommended for production use yet. See docs/status.md for the roadmap.

Upgrading from 0.1.0:

• Counters now live in counter families. incr on a Family::default() family fails with InvalidArgument; declare Family::counter(). Families in files written by 0.1.0 keep their 0.1.0 behavior.
• New file format. Files written by 0.2.0 use format version 2, which 0.1.0 can't open. Keep a backup if you may need to downgrade.
• Struct literals no longer compile for FamilyOptions and the model types (#[non_exhaustive]); use the setters.
• FUSE mounts are refused unless you opt in with allow_fuse(true).

The changelog has the steps.

What's new

• Counter families (D179, D186, D187), after Bigtable's aggregate families.
  • incr keeps a counter as one cell however often it is incremented.
  • incr_at and put_i64_at keep dated buckets, for hourly or daily totals.
  • Increments of one counter in one mutation or batch add up in order.
  • A delete restarts the counter from 0, and a bottommost compaction purges it.
• Large values. Values up to 4 GiB − 2 bytes (D188): a put above the inline limit goes to a blob file at commit time.
• Blob separation and blob GC. Values above a family's blob_threshold (default 4 KiB) move to blob files when flushed or compacted (D180). Blob GC follows the references each SST records (D184). backup copies separated values, and shrink relocates blob extents.
• Per-family compaction styles: Compaction::Tiered for write-heavy families, and Compaction::FifoByTime, which drops whole expired files on a timer (D168).
• zstd block compression with a per-family level (D175).
• A file at rest near its live size. After compact() and shrink(), a file above 5 MiB is 1.05–1.16× its live data, where it was 2–4× (D183, D185, D190).
• Flush-time version GC. A flush drops overwritten versions beyond max_versions and cells hidden at every snapshot, so hot rows reach disk already trimmed (D191). A bottommost compaction's purge runs under a guard against backfilled writes (D192).
• Custom merge operators work. Operators registered with Options::merge_operator reach the engine. A handle opened over an unregistered operator is read-only, as documented.
• The FUSE opt-in. Options::allow_fuse(true) accepts trusted local FUSE mounts (ntfs-3g, gocryptfs). Network filesystems are always refused (D173).
• Options::write_stall_timeout, and #[non_exhaustive] option and result structs with builder setters, so later fields are not breaking.

Performance

The Phase 2 gate (D193) uses phdb-bench sparse-wide --scale full: 10 shards, 64 MiB memtables, 256 MiB cache, one client thread. In the official runs 5–6 on a quiet 10-core Apple M5, Pigeonhole beats hand-keyed RocksDB on every measure and matches or beats SQLite EAV on throughput, get and put p99, and p99.9.

Store	ops/s	p99 µs	p99.9 µs	get p99	put p99	row-read p99	scan p99
Pigeonhole	28.7K / 30.4K	389 / 395	528 / 541	98 / 11	17 / 17	455 / 459	489 / 496
SQLite EAV	27.1K / 29.8K	245 / 250	1,040 / 938	156 / 152	178 / 172	222 / 231	293 / 307
RocksDB	20.4K / 20.4K	709 / 791	1,012 / 1,196	161 / 155	9 / 9	840 / 963	881 / 1,016

• The one documented gap is the p99 of reads of very wide, heavily overwritten rows: row reads about 2× SQLite EAV, scans about 1.6× (#387, Phase 3).
• The cause: an overwrite adds a version, and these reads step over the superseded versions until compaction removes them. A compacted hot row costs about the same per cell as SQLite.
• What to do now: the guide's Wide, overwritten rows says how to keep it down: read only the columns you need, set max_versions, and run compact() before read-heavy phases.

How it was tested

• Simulation. Deterministic simulation (SimVfs) with fault injection, process crashes and power loss, checked against a reference model with a record-level recovery oracle. The model runs exercise counter families, blob separation, large values and flush-time GC.
• Seed sweeps of 1–300 of the engine model check, tablets and public model suites on the changes that touch reads, writes, flushes and compaction.
• Other checks: Miri on the unsafe crates, loom on the concurrent ones, and CI on Linux, macOS and Windows.
• Performance regression checks. Every read- and write-path change was measured by CI's instruction counts (Linux callgrind) over hot-row, read and write shapes.
• D193's floors keep what Phase 2 reached. CI fails a change that makes any shape worse than its per-change threshold or puts it above its absolute ceiling. Every later official gate run, including the one before each release, is compared with the committed Phase 2 baseline (crates/bench/baselines/phase2-gate).
• Gate runs. The official sparse-wide gate runs 5–6, against SQLite EAV and RocksDB, on a quiet machine.

Known limits (tracked)

• Wide-row read tail: reads of wide, heavily overwritten rows have a p99 about 2× SQLite EAV for row reads and about 1.6× for scans. #387, Phase 3.
• Shard scaling: write throughput does not yet scale with shard count. #154, Phase 3.
• Open latency: about 14 ms against the 5 ms goal. #158, Phase 3.
• Application-owned mode still runs the default I/O backend's 2–16 I/O threads. Phase 3.
• The async API (async feature) is Phase 3.
• pigeonhole-cli (phdb) is a placeholder until Phase 4.

Crates

pigeonhole (start here), plus pigeonhole-engine, -format, -io, -pager, -wal, -memtable, -cache, -runtime, -shm, -sst, -compaction, -sim, -arrow and pigeonhole-cli, all at 0.2.0.

Docs

• Changelog: CHANGELOG.md, with migration notes.
• User guide: docs/guide covers getting started, concepts, data modeling (now with counter families and wide rows), durability, scans and filters, and errors. The agent reference is for coding agents.
• Design: the spec and the decisions, D168–D193 for Phase 2.

Pigeonhole 0.1.0 — Phase 1 core engine

Pigeonhole 0.1.0 is the first release: the Phase 1 core engine of an embedded, single-file, wide-column store in Rust (BigTable's data model with SQLite's deployment model).

Experimental 0.x. The core engine is complete and fault-tested in simulation, but the on-disk format and the API may change before 1.0 (FORMAT.md), the wide-column model (Phase 2) and latency work (Phase 3) are still to come, and it is not recommended for production use yet. See docs/status.md for the roadmap.

What's in it

• One file, embedded. A database is a single .phdb file (plus WAL sidecars and a shared-memory region while open). No server, zero required configuration; Options::default() is valid.
• Thread-per-core shards with per-shard WAL streams and group commit; durability levels None, Buffered, GroupSync, Sync.
• Cross-shard commits are all-or-nothing (two-phase commit), with a global visibility watermark and MVCC snapshots.
• Tablets split, merge and move between shards under load and size, on by default.
• Per-family LSM: memtables in a shared-memory arena, SSTs, leveled compaction, HBase-style delete and GC semantics, counters (incr) and merge operators, TTL.
• Reader processes attach read-only through shared memory; snapshots expire cleanly across a writer restart (SnapshotExpired).
• Maintenance: flush, compact, backup, shrink (now returns space to the filesystem).
• Cheap opens: about 14 ms and well under 1 MiB written for a small database, regardless of shard count.
• Clear errors through a flat, stable ErrorCode (C-ABI friendly), including Busy (retryable stall), BatchTooLarge, ShmUnavailable with sizing advice, WouldDeadlock.

How it was tested

• Deterministic simulation (SimVfs) with fault injection, process crashes and power loss, checked against a reference model with a record-level recovery oracle.
• Seed sweeps of 1–300 on every suite before the release, including a mode that keeps I/O in flight across scheduling points; Miri on the unsafe crates; loom on the concurrent ones; CI on Linux, macOS and Windows.
• A pre-release edge-case review (#90) of busy-waits, embedding modes, platforms, error paths, resources, concurrency, defaults and documentation, whose findings were all fixed or scheduled.

Known limits (tracked)

• Write throughput does not yet scale with shard count (efficiency 0.10–0.49 at 2–8 shards on a 10-core Mac; commit-latency and visibility-bound) — #154, Phase 3.
• Open latency ~14 ms against the 5 ms goal — #158, Phase 3.
• File at rest can be 2–4× live data because of power-of-two extents — #185, Phase 2.

Crates

pigeonhole (start here), plus pigeonhole-engine, -format, -io, -pager, -wal, -memtable, -cache, -runtime, -shm, -sst, -compaction, -sim, -arrow and pigeonhole-cli.

Docs

• User guide: docs/guide (getting started, concepts, data modeling, durability, scans and filters, errors) and an agent reference for coding agents.
• Design: spec and decisions.