Skip to content

Pigeonhole 0.1.0 — Phase 1 core engine

Pre-release
Pre-release

Choose a tag to compare

View all tags
@CodingAnarchy CodingAnarchy released this 08 Oct 06:44
· 361 commits to main since this release
v0.1.0
3abef59
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Pigeonhole 0.1.0 is the first release: the Phase 1 core engine of an embedded, single-file, wide-column store in Rust (BigTable's data model with SQLite's deployment model).

Experimental 0.x. The core engine is complete and fault-tested in simulation, but the on-disk format and the API may change before 1.0 (FORMAT.md), the wide-column model (Phase 2) and latency work (Phase 3) are still to come, and it is not recommended for production use yet. See docs/status.md for the roadmap.

What's in it

  • One file, embedded. A database is a single .phdb file (plus WAL sidecars and a shared-memory region while open). No server, zero required configuration; Options::default() is valid.
  • Thread-per-core shards with per-shard WAL streams and group commit; durability levels None, Buffered, GroupSync, Sync.
  • Cross-shard commits are all-or-nothing (two-phase commit), with a global visibility watermark and MVCC snapshots.
  • Tablets split, merge and move between shards under load and size, on by default.
  • Per-family LSM: memtables in a shared-memory arena, SSTs, leveled compaction, HBase-style delete and GC semantics, counters (incr) and merge operators, TTL.
  • Reader processes attach read-only through shared memory; snapshots expire cleanly across a writer restart (SnapshotExpired).
  • Maintenance: flush, compact, backup, shrink (now returns space to the filesystem).
  • Cheap opens: about 14 ms and well under 1 MiB written for a small database, regardless of shard count.
  • Clear errors through a flat, stable ErrorCode (C-ABI friendly), including Busy (retryable stall), BatchTooLarge, ShmUnavailable with sizing advice, WouldDeadlock.

How it was tested

  • Deterministic simulation (SimVfs) with fault injection, process crashes and power loss, checked against a reference model with a record-level recovery oracle.
  • Seed sweeps of 1–300 on every suite before the release, including a mode that keeps I/O in flight across scheduling points; Miri on the unsafe crates; loom on the concurrent ones; CI on Linux, macOS and Windows.
  • A pre-release edge-case review (#90) of busy-waits, embedding modes, platforms, error paths, resources, concurrency, defaults and documentation, whose findings were all fixed or scheduled.

Known limits (tracked)

  • Write throughput does not yet scale with shard count (efficiency 0.10–0.49 at 2–8 shards on a 10-core Mac; commit-latency and visibility-bound) — #154, Phase 3.
  • Open latency ~14 ms against the 5 ms goal — #158, Phase 3.
  • File at rest can be 2–4× live data because of power-of-two extents — #185, Phase 2.

Crates

pigeonhole (start here), plus pigeonhole-engine, -format, -io, -pager, -wal, -memtable, -cache, -runtime, -shm, -sst, -compaction, -sim, -arrow and pigeonhole-cli.

Docs

Assets 2
Loading