diff --git a/docs/plans/2026-06-22-transaction-module-extraction.md b/docs/plans/2026-06-22-transaction-module-extraction.md new file mode 100644 index 0000000..4362668 --- /dev/null +++ b/docs/plans/2026-06-22-transaction-module-extraction.md @@ -0,0 +1,311 @@ +# transaction.rs Module Extraction — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Split the 5,377-line `src/transaction.rs` god-module into a `src/transaction/` module and extract five cohesive units (SlotPacker, FreemapRecycle, FaultInjector, CommitProtocol, StagingTxn) behind narrow interfaces — a pure behavior-preserving refactor. + +**Architecture:** Six incremental steps, low-risk → high-risk, each its own commit/PR and each leaving the full suite green. State sub-structs own only their field cluster and take `&mut PageCache` + an `alloc` closure; behavior units operate over a context. The existing ~2,281-line test suite is the oracle — no test *behavior* changes. + +**Tech Stack:** Rust, `cargo test` / `cargo clippy --workspace --all-targets -- -D warnings` / `cargo fmt`, `maturin develop && pytest` (Python binding unaffected but engine changes are re-verified). + +**Spec:** `docs/specs/2026-06-22-transaction-module-extraction-design.md` (read it first). + +**Branch:** `feature/transaction-extraction` carries the spec + this plan. **Step 1 must branch off `main` AFTER #75 (abort-leak) merges** — #75 edits `transaction.rs`, and a file-move-vs-edit conflict is unmanageable. Each later step branches off updated `main`. + +--- + +## The refactor discipline (applies to EVERY task) + +This is a PURE refactor: **no behavior change, no API change, no on-disk format change.** Therefore: +- **You do not write new failing tests.** The oracle is the existing suite. The "test" step for every task is: run the FULL suite and confirm it stays green (same pass count, zero failures), prove behavior was preserved. +- **The green gate for every task** (run all four; all must pass before commit): + - `cargo test` (NOT `--lib` — the integration tests in `tests/` are part of the oracle) + - `cargo clippy --workspace --all-targets -- -D warnings` + - `cargo fmt --check` + - `cd python && source .venv/bin/activate && maturin develop && python -m pytest -q` (the binding is unaffected, but the engine moved — re-verify) +- **If a step changes a test's observed behavior** (a different error, a different value, a panic) that is a REFACTOR BUG, not a test that needs updating — fix the code. The ONLY legitimate test-code changes are: (a) re-pathing/`use` updates from the file-split, and (b) the fault-flag plumbing moving to `FaultInjector` (Task 2). +- **NO Claude/AI/Anthropic references** in commits/comments. Comments explain WHY; preserve the existing rich invariant comments verbatim when moving code (do not paraphrase or drop them). +- **Per-unit surface check:** after each extraction, grep for direct access to the extracted struct's fields from outside its module; there should be none (access goes through the interface). + +--- + +### Task 1: File-split into `src/transaction/` + +Pure code movement. The `TransactionManager` struct, its fields, and every method body are **unchanged** — they just move into concern files. This is the foundation that makes every later diff small. + +**Files:** +- Create: `src/transaction/mod.rs` (the struct, `Roots`, `Savepoint`, the free fns `cow_alloc`/`structural_extend`, module `use`s, and `pub(crate) mod` declarations for the submodules) +- Create: `src/transaction/recovery.rs`, `lifecycle.rs`, `commit.rs`, `staging.rs`, `freemap.rs`, `packing.rs`, `savepoints.rs`, `named_roots.rs`, `read.rs`, `mutate.rs`, `config.rs`, `stats.rs`, `tests.rs` +- Delete: `src/transaction.rs` (its content is distributed) +- Modify: `src/lib.rs` (the `mod transaction;` line is unchanged — a directory module resolves the same) + +- [ ] **Step 1: Create the module skeleton.** Make `src/transaction/` and move the WHOLE current `src/transaction.rs` into `src/transaction/mod.rs` unchanged (so it compiles identically). Add nothing yet. Run `cargo build` — must compile. This isolates the directory-rename from the content-move. + +- [ ] **Step 2: Move impl blocks into concern files, one file at a time.** For each concern file, cut the relevant `impl TransactionManager { ... }` methods (and any concern-private free fns) out of `mod.rs` and into the file as `impl TransactionManager { ... }` (the child module sees the parent's private fields — no `pub` needed). Add `mod ;` to `mod.rs`. The method→file mapping (by the method names in `mod.rs`): + - `recovery.rs`: `create_new`, `open_existing` + - `lifecycle.rs`: `begin`, `begin_inner`, `commit`, `commit_inner`, `rollback`, `rollback_inner`, `check_alive`, `poison_on_fatal`, `is_poisoned`, `force_poison_for_test`, `is_active` + - `commit.rs`: (leave `commit_inner` in `lifecycle.rs` for now — Task 5 extracts it here) + - `staging.rs`: `allocate`, `allocate_tagged`, `allocate_inner`, `membership_insert_candidate`, `handle_table_insert_candidate`, `abort_allocate_prepare`, `membership_remove_candidate`, `inject_membership_failure` + - `freemap.rs`: `take_freemap_tree`, `put_freemap_tree`, `allocate_data_page`, `ht_insert`, `freemap_mark_free_committed_path`, `persist_freemap`, `reclaim_freemap_orphans`, `cache_watermark`, and the free fns `cow_alloc` + `structural_extend` (move them here; they are `fn` not methods — keep them module-private, `pub(super)` if `mod.rs` or another file needs them — `ht_insert` uses `cow_alloc`) + - `packing.rs`: `release_data_slot`, `ensure_handle_table`, `insert_into_data_page` + - `savepoints.rs`: `savepoint`, `savepoint_inner`, `rollback_to`, `rollback_to_inner`, `release`, `release_inner` + - `named_roots.rs`: `encode_root_name`, `set_root_name`, `set_root_name_inner`, `get_root_name`, `get_root_name_inner`, `clear_root_name`, `clear_root_name_inner` + - `read.rs`: `read`, `read_inner`, `tag`, `tag_inner`, `lookup_live`, `live_handle_table_root`, `client_byte`, `client_byte_inner`, `set_client_byte`, `set_client_byte_inner`, `handles_with_tag`, `handles_with_tag_inner`, `handles`, `handles_inner` + - `mutate.rs`: `update`, `update_inner`, `delete`, `delete_inner`, `delete_tagged`, `delete_tagged_inner`, `delete_with_tag`, `delete_with_tag_inner`, `delete_many`, `delete_many_inner` + - `config.rs`: `set_cache_max_bytes`, `set_spillway_max_bytes`, `set_drain_insertion` + - `stats.rs`: `counters`, `spillway_capacity`, `file_page_count`, `sparse_data_pages`, `sparse_data_pages_inner`, `data_page_ids_snapshot`, `handle_live_page_id`, `handle_live_page_id_inner`, `current_handle_table_root_page`, `test_forge_freemap_orphan`, `test_forge_corrupt_dead_page` + - `tests.rs`: the entire `#[cfg(test)] mod tests { ... }` block. Becomes `#[cfg(test)] mod tests;` in `mod.rs` and the file content is `use super::*;` + the test bodies. (If a single tests.rs is unwieldy, splitting per concern is allowed but optional — do it only if it falls out cleanly.) + - After each file move: `cargo build`. Fix only `use`/visibility fallout (e.g. a concern-private helper another file now needs becomes `pub(super)`). + +- [ ] **Step 3: Resolve `use` and visibility.** `mod.rs` keeps the top-level `use` imports needed broadly; each concern file adds `use super::*;` (or specific imports). The free fns `cow_alloc`/`structural_extend` and any helper a sibling module calls become `pub(super)` (crate-internal, module-scoped). Run `cargo build` clean. + +- [ ] **Step 4: Green gate.** Run all four checks (see discipline). The test count must equal the pre-split count exactly. `git diff main --stat` should show only moves (line counts roughly conserved) — no logic change. + +- [ ] **Step 5: Commit.** `refactor: split transaction.rs into a transaction/ module by concern` + +> **PR boundary:** Task 1 + Task 2 ship as **PR 1** (both low-risk). The remaining tasks are one PR each. + +--- + +### Task 2: Extract `FaultInjector` (`#[cfg(test)]`) + +Consolidate the four test-only `Cell` flags into one test-only struct, off the production type. + +**Files:** +- Create: `src/transaction/fault.rs` +- Modify: `src/transaction/mod.rs` (the struct's four `#[cfg(test)]` fields → one), `src/transaction/staging.rs` + `mutate.rs` (the `inject_*` consult sites), `src/transaction/tests.rs` (tests that arm the flags) + +- [ ] **Step 1: Define `FaultInjector`** in `fault.rs`: + +```rust +//! Test-only fault injection consolidated off the production TransactionManager +//! (review 2026-06-22 SMELL #4). Each Cell arms a one-shot or countdown failure +//! at a precise commit-protocol divergence window; see the BUG#2 staging tests. +use std::cell::Cell; + +#[cfg(test)] +#[derive(Default)] +pub(super) struct FaultInjector { + pub fail_next_membership_op: Cell, + pub fail_next_handle_table_op: Cell, + pub fail_next_update_value_write: Cell, + pub fail_membership_op_after: Cell, +} +``` + +- [ ] **Step 2: Replace the four struct fields** in `mod.rs` with one (keep the existing field doc comments, relocated to `fault.rs` or condensed): + +```rust + #[cfg(test)] + fault: fault::FaultInjector, +``` +and `#[cfg(test)] fault: fault::FaultInjector::default(),` in BOTH constructors (`create_new`, `open_existing`). + +- [ ] **Step 3: Rewire the consult/arm sites.** Every `self.fail_next_membership_op` → `self.fault.fail_next_membership_op`, etc. (production consult sites are already behind `#[cfg(test)]`; the existing `inject_membership_failure` helper and `allocate_inner`'s `#[cfg(test)]` branches). In `tests.rs`, every `tm.fail_next_membership_op.set(true)` → `tm.fault.fail_next_membership_op.set(true)`. (grep `fail_next_|fail_membership_op_after` to find all sites.) + +- [ ] **Step 4: Green gate** (all four). Crucially confirm `cargo build --release` compiles (the cfg(test) field is fully absent in release). + +- [ ] **Step 5: Commit.** `refactor: consolidate test fault flags into a cfg(test) FaultInjector` + +--- + +### Task 3: Extract `SlotPacker` + +Owned state struct for the R1 slot-packing cluster. + +**Files:** +- Modify: `src/transaction/packing.rs` (define `SlotPacker` + move the packing logic), `src/transaction/mod.rs` (replace 3 fields with `packer: SlotPacker`), `src/transaction/lifecycle.rs` (begin/commit/rollback delegate), `src/transaction/savepoints.rs` (snapshot/restore through the interface), `src/transaction/staging.rs` + `mutate.rs` (call sites) + +- [ ] **Step 1: Define `SlotPacker`** in `packing.rs`, owning the three fields and the packing logic: + +```rust +pub(super) struct SlotPacker { + committed_live_slots: FxHashMap, + current_live_slots: FxHashMap, + insert_cursor: Option, +} + +impl SlotPacker { + pub(super) fn new() -> Self { /* empty maps, cursor None */ } + // R1 packing: append `value` to the cursor page (or a fresh page from + // `alloc`), updating live-slot counts. Returns (page_id, slot). Moves the + // body of TransactionManager::insert_into_data_page here verbatim, with + // field accesses self.* -> the struct's fields and the data-page allocation + // delegated to the `alloc` closure (formerly self.allocate_data_page()). + pub(super) fn insert( + &mut self, + cache: &mut PageCache, + alloc: &mut dyn FnMut(&mut PageCache) -> Result, + value: &[u8], + ) -> Result<(u64, u16)>; + // Decrement the live-slot count for a released slot (the body of + // release_data_slot). Drops cursor bookkeeping consistently. + pub(super) fn release(&mut self, page_id: u64); + // Lifecycle, matching the current begin/commit/rollback handling of + // current_live_slots + insert_cursor: + pub(super) fn begin(&mut self); // current = committed.clone(); cursor = None + pub(super) fn commit(&mut self); // committed = current.clone() (or take) + pub(super) fn rollback(&mut self); // current = committed.clone(); cursor = None + // Savepoint snapshot/restore (Savepoint already captures live_slots + cursor): + pub(super) fn snapshot(&self) -> (FxHashMap, Option); + pub(super) fn restore(&mut self, snap: (FxHashMap, Option)); + // Read accessors the stats/introspection paths need (current counts): + pub(super) fn current_live_slots(&self) -> &FxHashMap; + pub(super) fn is_current_empty(&self) -> bool; +} +``` + +- [ ] **Step 2: Replace the three fields** in `mod.rs` with `packer: SlotPacker`, init `SlotPacker::new()` in both constructors. + +- [ ] **Step 3: Move `insert_into_data_page` and `release_data_slot` bodies into `SlotPacker::insert`/`release`.** Where `insert_into_data_page` called `self.allocate_data_page()`, the caller now passes that as the `alloc` closure: in the staging/mutate call sites, `self.packer.insert(&mut cache, &mut |c| self.allocate_data_page_for(c), value)` — but `self.allocate_data_page` borrows `self.freemap`-state, disjoint from `self.packer`, so build the closure with the freemap fields borrowed as locals (the existing disjoint-borrow pattern; see `ht_insert`). The `ensure_handle_table` method stays on the manager (it touches the handle table + roots, not the packer). + +- [ ] **Step 4: Delegate lifecycle + savepoints.** In `lifecycle.rs`, `begin_inner`/`commit_inner`/`rollback_inner`'s handling of `current_live_slots`/`committed_live_slots`/`insert_cursor` becomes `self.packer.begin()/commit()/rollback()`. In `savepoints.rs`, where `Savepoint` is built/restored, use `self.packer.snapshot()` / `self.packer.restore(...)` instead of cloning the fields directly. The `Savepoint` struct keeps its `live_slots`/`insert_cursor` fields (they hold the snapshot tuple's parts). + +- [ ] **Step 5: Rewire stats/introspection** reads of `current_live_slots` to `self.packer.current_live_slots()` and the `tm.current_live_slots.is_empty()` test assertions to `tm.packer.is_current_empty()` (or a test accessor). grep `current_live_slots|committed_live_slots|insert_cursor` to find all sites. + +- [ ] **Step 6: Green gate** (all four) + surface check (no external access to `SlotPacker`'s private fields). + +- [ ] **Step 7: Commit.** `refactor: extract SlotPacker (R1 live-slot packing) as an owned unit` + +--- + +### Task 4: Extract `FreemapRecycle` (the hardest) + +Owned state struct for the structural-recycle cluster + the freemap commit/alloc paths. + +**Files:** +- Modify: `src/transaction/freemap.rs` (define `FreemapRecycle` + move the freemap methods), `src/transaction/mod.rs` (replace 5 fields), `src/transaction/lifecycle.rs` (recycle lifecycle), `src/transaction/savepoints.rs` (rollback interaction), call sites in `staging.rs`/`mutate.rs`/`stats.rs` + +- [ ] **Step 1: Define `FreemapRecycle`** in `freemap.rs`, owning the five fields: + +```rust +pub(super) struct FreemapRecycle { + hint: u64, + structural_reuse: Vec, + structural_superseded: Vec, + pending_structural_frees: Vec, + session_owned: FxHashSet, +} +``` + +- [ ] **Step 2: Move the freemap machinery into `impl FreemapRecycle`.** Move the bodies of `take_freemap_tree`, `put_freemap_tree`, `structural_extend` (the free fn), and `cow_alloc`'s freemap portion into methods on `FreemapRecycle` that take `&mut PageCache` and the roots' `{freemap_page, freemap_depth}` (mutating them via out-params or by taking `&mut Roots`). Interface: + +```rust +impl FreemapRecycle { + pub(super) fn new() -> Self; + // Allocate a page id for COW work: reuse a free bit (drawing structural COW + // targets from the recycle pool, never the bitmap) or extend. Updates the + // roots' freemap_page/depth and the hint; accumulates supersedes internally. + // (Body = today's cow_alloc + take/put_freemap_tree dance, now self-contained.) + pub(super) fn allocate(&mut self, cache: &mut PageCache, roots: &mut Roots, reuse_enabled: bool) -> Result; + // Mark a page free through the committed-path COW (body of + // freemap_mark_free_committed_path). + pub(super) fn mark_free_committed_path(&mut self, cache: &mut PageCache, roots: &mut Roots, id: u64) -> Result<()>; + // Commit-time persist (body of persist_freemap): apply txn_freed_pages, COW + // the touched leaves+spine via the pool, promote structural_superseded. + pub(super) fn persist(&mut self, cache: &mut PageCache, roots: &mut Roots, txn_freed_pages: &[u64]) -> Result<()>; + // Defrag orphan sweep (body of reclaim_freemap_orphans), with the + // savepoints-active guard passed in. + pub(super) fn reclaim_orphans(&mut self, cache: &mut PageCache, roots: &Roots, savepoint_active: bool, superblock_count: u32) -> Result; + // Lifecycle: + pub(super) fn begin(&mut self); // structural_reuse = pending_structural_frees.clone(); session_owned.clear() + pub(super) fn commit(&mut self); // promote structural_superseded + leftover reuse -> pending_structural_frees + pub(super) fn rollback(&mut self); // structural_reuse back to pending; superseded.clear(); session_owned.clear() + // The reuse pool, for the orphan-sweep exclusion set (read-only): + pub(super) fn pool_ids(&self) -> impl Iterator + '_; +} +``` + +- [ ] **Step 3: Replace the five fields** in `mod.rs` with `freemap: FreemapRecycle`, init `FreemapRecycle::new()` in both constructors. The `cow_alloc` free fn either becomes a `FreemapRecycle` method or a thin wrapper delegating to `self.freemap.allocate(...)`. + +- [ ] **Step 4: Rewire the THREE allocation call sites** (`allocate_data_page`/`ht_insert`'s closure, the membership insert/remove sites in `staging.rs`). Each currently captures `let hint = &mut self.freemap_hint; let pool = &mut self.structural_reuse;` etc. — now they borrow `&mut self.freemap` (one field, disjoint from `self.handle_table`/`self.membership_index`/`self.cache`). Confirm the disjoint-borrow still satisfies the checker (it should — one field vs the structure handles). Update `allocate_data_page`, `ht_insert`, and the membership candidate/remove methods. + +- [ ] **Step 5: Delegate lifecycle + rollback interaction.** `begin_inner`/`commit_inner`/`rollback_inner` call `self.freemap.begin()/commit()/rollback()` instead of the inline stream handling. The `reclaim_freemap_orphans` exclusion set uses `self.freemap.pool_ids()`. `persist_freemap` call in commit becomes `self.freemap.persist(&mut cache, &mut self.current_roots, &self.txn_freed_pages)`. + +- [ ] **Step 6: Green gate** (all four — the recycle pins `structural_recycle_one_commit_defer`/`..rollback_resets_pools`/`..no_lost_or_double_free` and the orphan-sweep/savepoint tests are the load-bearing oracle here; they MUST stay green) + surface check. + +- [ ] **Step 7: Commit.** `refactor: extract FreemapRecycle (structural recycle + persist/reclaim) as an owned unit` + +--- + +### Task 5: Extract `CommitProtocol` + +The `commit_inner` sequence into a behavior unit (function-module over a context; promote to an owned struct only if it reads cleanly). + +**Files:** +- Modify: `src/transaction/commit.rs` (the CommitProtocol unit), `src/transaction/lifecycle.rs` (`commit_inner` delegates) + +- [ ] **Step 1: Move the `commit_inner` body into `commit.rs`** as a function (or `CommitProtocol::run`) taking the context it needs: + +```rust +// The 3-fsync commit sequence (I28 pre-drain flush -> FreemapRecycle::persist -> +// data fsync -> superblock build/write/fsync -> roots promotion). The +// data-fsync-before-superblock-fsync ordering and I18 allocate-before-merge are +// preserved verbatim. Operates over the manager's parts; owns no state. +pub(super) fn run_commit( + cache: &RefCell, + committed_roots: &mut Roots, + current_roots: &mut Roots, + freemap: &mut FreemapRecycle, + packer: &mut SlotPacker, + txn_freed_pages: &mut Vec, + txn_counter: &mut u64, + superblock_count: u32, +) -> Result<()>; +``` +(Exact parameter set = whatever `commit_inner` touches; thread each as `&mut`/`&` rather than `self`. If the param list is unwieldy, a small `CommitCtx<'a>` struct bundling the `&mut`s is allowed.) + +- [ ] **Step 2: `commit_inner` (lifecycle.rs) becomes a thin caller** of `run_commit(...)`, passing its fields. The poison-on-fatal wrapper (`commit` → `poison_on_fatal(commit_inner())`) stays on the manager. + +- [ ] **Step 3: Green gate** (all four — the fsync-count integration test `tests/spillway_integration.rs` asserting `fsync_delta == 3` and the `persist_freemap_does_not_reuse_committed_live_pages` I18 guardrail are the load-bearing oracle). + +- [ ] **Step 4: Commit.** `refactor: extract the commit protocol (3-fsync sequence) into commit.rs` + +--- + +### Task 6: Extract `StagingTxn` + +The BUG#2 atomic staging of `allocate_inner` into a behavior unit. Last (touches the most units). + +**Files:** +- Modify: `src/transaction/staging.rs` (the StagingTxn unit), and the `allocate_inner` caller + +- [ ] **Step 1: Move the staging into `staging.rs`** as a function/unit taking the context: the cache, both roots, `&mut HandleTable`, `&mut MembershipIndex`, `&mut FreemapRecycle`, `&mut SlotPacker`, `&mut txn_freed_pages`, and (test-only) `&FaultInjector`. Move the bodies of `handle_table_insert_candidate`, `membership_insert_candidate`, `membership_remove_candidate`, `abort_allocate_prepare`, and the PREPARE/INSTALL flow of `allocate_inner` here. Preserve verbatim: the compute-without-install discipline, the local `ht_freed`/`mi_freed` lists appended to `txn_freed_pages` only in the INSTALL phase, the bounded-residue-on-abort contract (documented in PR #75), and the `#[cfg(test)]` fault hooks (now via `&FaultInjector`). + +```rust +pub(super) fn run_allocate( + ctx: &mut StagingCtx<'_>, // bundles cache, roots, handle_table, membership_index, freemap, packer, txn_freed_pages + value: &[u8], + tag: u32, + #[cfg(test)] fault: &FaultInjector, +) -> Result; +``` + +- [ ] **Step 2: `allocate_inner` becomes a thin caller** assembling the context and delegating. `allocate`/`allocate_tagged` (the public wrappers with `check_alive` + `poison_on_fatal`) stay on the manager. + +- [ ] **Step 3: Green gate** (all four — the staging oracle: `allocate_membership_failure_leaves_maps_consistent`, `allocate_handle_table_failure_leaves_maps_consistent`, `aborted_tagged_allocate_with_freemap_reuse_is_consistent_and_rollback_reclaims`, and the BUG#2 atomic-staging tests MUST stay green) + surface check. + +- [ ] **Step 4: Commit.** `refactor: extract StagingTxn (BUG#2 atomic prepare/install) into staging.rs` + +- [ ] **Step 5: Final whole-refactor review.** After all six, dispatch an adversarial review of `git diff ..HEAD` confirming: zero behavior change (the suite is the proof), each unit's surface is narrow, the durability invariant comments survived the moves, and no field is `pub` that wasn't. Then `transaction/mod.rs` should be a slim orchestrator + struct, and the largest concern file should be a fraction of the original 5,377. + +--- + +## Self-Review (against the spec) + +**Spec coverage:** +- Module split by concern → Task 1. ✅ +- State sub-structs own only their cluster + closures → Tasks 3 (SlotPacker), 4 (FreemapRecycle). ✅ +- Behavior units over a context + function-module fallback → Tasks 5 (CommitProtocol), 6 (StagingTxn); the `CommitCtx`/`StagingCtx` bundle is the function-module form. ✅ +- `#[cfg(test)]` FaultInjector (not trait object) → Task 2. ✅ +- Savepoint snapshot/restore through the interface → Task 3 Step 4. ✅ +- Low-risk-first order, each green → Tasks 1-6 in order; the green gate in every task. ✅ +- Reads/mutations/named-roots/stats/config move to files but stay `impl TransactionManager` (no sub-struct) → Task 1's mapping; not extracted in 3-6. ✅ +- #75-merge dependency → header + branch note. ✅ +- Pure behavior-preserving (no new test behavior) → the refactor discipline section. ✅ + +**Placeholder scan:** the interface signatures are marked "illustrative"/"exact = whatever the method touches" where the precise param set is a mechanical read of the existing body — that is concrete guidance for a refactor (the bodies already exist), not a placeholder. No `TODO`/`TBD`. + +**Type consistency:** `SlotPacker` (`insert`/`release`/`begin`/`commit`/`rollback`/`snapshot`/`restore`); `FreemapRecycle` (`allocate`/`mark_free_committed_path`/`persist`/`reclaim_orphans`/`begin`/`commit`/`rollback`/`pool_ids`); `FaultInjector` (the four `Cell`s); `run_commit`/`CommitCtx`; `run_allocate`/`StagingCtx`. Consistent across tasks. + +> **Open risk (carried from the spec):** if an owned-struct extraction (esp. FreemapRecycle, Task 4) hits a borrow wall, fall back to the function-module form (decision 4) in the same file with the same narrow interface — do not force ownership. Note the choice in that task's commit message. diff --git a/docs/specs/2026-06-22-transaction-module-extraction-design.md b/docs/specs/2026-06-22-transaction-module-extraction-design.md new file mode 100644 index 0000000..cc7e91b --- /dev/null +++ b/docs/specs/2026-06-22-transaction-module-extraction-design.md @@ -0,0 +1,248 @@ +# transaction.rs Module Extraction — Design Spec + +Status: approved 2026-06-22 +Type: pure behavior-preserving refactor (no on-disk format change, no API change, +no behavior change). +Origin: the last carve-out of the 2026-06-22 fresh-eyes review (SMELL #4, +`transaction.rs:1-2611` god-module). + +--- + +## Summary + +`src/transaction.rs` is 5,377 lines (~3,096 production, ~2,281 tests) — by far the +largest file in the crate — holding the entire `TransactionManager`: the commit +protocol, savepoints, the R1 slot-packer, the freemap recycle machinery, the +BUG#2 atomic staging, named roots, reads, mutations, the poison flag, and four +`#[cfg(test)]` fault-injection flags baked into the production struct. Every +durability invariant (3-fsync ordering, I18 freemap window, BUG#2 atomic staging, +R1 cursor accounting, watermark rollback, the one-commit structural-recycle +defer) is encoded as prose cross-references rather than types or module +boundaries, so a reviewer must hold all of them simultaneously to safely change +the commit path. + +This refactor splits the file into a `src/transaction/` module and extracts +cohesive units behind narrow interfaces: three **owned state sub-structs** +(`SlotPacker`, `FreemapRecycle`, `FaultInjector`) and two **behavior units** +(`CommitProtocol`, `StagingTxn`). It is **purely structural** — the byte-for-byte +behavior, the public API, and the on-disk format are unchanged, and the existing +~2,281-line test suite is the oracle that proves it. + +## Motivation + +- The commit path is "the single biggest barrier to safely changing" (review): + a one-line change requires understanding the freemap recycle, the slot packer, + the staging, and the fsync ordering at once. +- Test concerns leak into the core type: four `#[cfg(test)]` `Cell` fields sit on + the production `TransactionManager`. +- The freemap work (PRs #70/#71) added ~600 lines and five recycle fields, + growing the module further. + +## Design decisions + +1. **Module split by concern, not by layer.** `src/transaction.rs` → + `src/transaction/` with `mod.rs` holding the `TransactionManager` struct, + `Roots`, and `Savepoint`, and impl blocks moved into focused submodule files. + Rust privacy is module-scoped *and inherited by descendants*, so a child + module (`transaction::freemap`) can access the parent struct's private fields + with zero visibility churn — no field is made `pub` that wasn't. + +2. **State sub-structs own ONLY their cluster; methods take shared resources as + parameters.** `SlotPacker`, `FreemapRecycle`, `FaultInjector` hold just their + fields. Their methods take `&mut PageCache` (the caller holds the `RefMut` + from `self.cache.borrow_mut()`) and, where they allocate, an `alloc`/`extend` + **closure** — the exact disjoint-field-borrow + closure pattern the freemap + integration already uses (`take_freemap_tree`/`put_freemap_tree`, the + `cow_alloc` closure). This pattern is already proven to satisfy the borrow + checker against `self.handle_table` / `self.cache` simultaneously. + +3. **Behavior units own almost nothing; they operate over a context.** + `CommitProtocol` and `StagingTxn` are thin — they take the cache, the roots, + and `&mut` to the relevant sub-structs (per the review's "CommitProtocol over + a roots+freemap snapshot"), closer to function-modules-with-a-context than + owned objects. + +4. **"Owned struct vs free-function module" is decided per unit DURING execution, + by what compiles cleanly.** If extracting a unit as an owned struct forces + contortions (an orchestrator needing simultaneous `&mut` to three sub-structs + through `self`), the fallback is a free-function module in the same file with + the same narrow interface — no forced ownership. This is an explicit, + pre-blessed fallback, not a failure. + +5. **`FaultInjector` is `#[cfg(test)]`, not a trait object.** The manager holds a + single `#[cfg(test)] fault: FaultInjector` (one field replacing four), and + production code consults it only behind `#[cfg(test)]`. A `dyn` trait object + would add a vtable + an always-present `Option>` for a test-only + concern — more machinery than the smell warrants. + +6. **Incremental, low-risk-first ordering, each step green.** Six steps: + file-split → FaultInjector → SlotPacker → FreemapRecycle → CommitProtocol → + StagingTxn. The risky commit/staging extractions land last, on an + already-organized base, as small diffs. + +## Module structure + +``` +src/transaction/ + mod.rs TransactionManager struct, Roots, Savepoint, field docs, + the public-API wrappers that delegate to the units + recovery.rs create_new, open_existing + lifecycle.rs begin/commit/rollback (the *_inner orchestration that calls + CommitProtocol) + commit.rs CommitProtocol (the 3-fsync sequence + superblock write) + staging.rs StagingTxn (BUG#2 candidates, abort_allocate_prepare, install) + freemap.rs FreemapRecycle (the recycle pool + persist_freemap + + reclaim_freemap_orphans + take/put_freemap_tree) + packing.rs SlotPacker (R1 live-slots + insert_cursor) + savepoints.rs savepoint / rollback_to / release + named_roots.rs set/get/clear_root_name + encode_root_name + read.rs read, tag, client_byte, handles, handles_with_tag, lookups + mutate.rs update, delete, delete_tagged, delete_with_tag, delete_many + config.rs set_cache_max_bytes / set_spillway_max_bytes / set_drain_insertion + stats.rs counters, spillway_capacity, file_page_count, sparse_data_pages, + data_page_ids_snapshot, handle_live_page_id, introspection + fault.rs #[cfg(test)] FaultInjector + tests.rs the test suite (or per-concern test submodules) +``` + +## The units + +### `SlotPacker` (packing.rs) — owned state +- **State:** `committed_live_slots: FxHashMap`, `current_live_slots: + FxHashMap`, `insert_cursor: Option`. +- **Owns:** R1 slot packing — `insert_into_data_page`, `release_data_slot`, the + live-slot accounting and the insert cursor. +- **Interface (illustrative):** + - `insert(&mut self, cache, alloc: impl FnMut(&mut PageCache)->Result, value) -> Result<(u64, u16)>` + - `release_slot(&mut self, page_id: u64)` + - lifecycle: `begin(&mut self)` (clone committed→current, reset cursor), + `commit(&mut self)` (promote current→committed), `rollback(&mut self)` + (reset current←committed). + - savepoint hooks: `snapshot() -> (FxHashMap, Option)` and + `restore(snap)` — because `Savepoint` already captures `live_slots` + + `insert_cursor`. +- **Depends on:** `PageCache` + an allocator closure (the data page comes from the + freemap path). No reach into the freemap internals. + +### `FreemapRecycle` (freemap.rs) — owned state (the hardest) +- **State:** `freemap_hint: u64`, `structural_reuse: Vec`, + `structural_superseded: Vec`, `pending_structural_frees: Vec`, + `freemap_session_owned: FxHashSet`. +- **Owns:** `take_freemap_tree` / `put_freemap_tree`, the structural `extend` + (pool-then-file), `allocate_data_page`'s freemap path, `cow_alloc`, + `freemap_mark_free_committed_path`, `persist_freemap`, + `reclaim_freemap_orphans`, and the recycle lifecycle (begin seeds + `structural_reuse` from `pending_structural_frees`; commit promotes + `structural_superseded`; rollback restores). +- **Interface:** an `allocate(cache, roots, reuse_enabled) -> Result` that + yields a reusable-or-extended page id and threads the tree handle internally; + `mark_free_committed_path`, `persist(cache, roots)`, `reclaim_orphans(cache, + roots, savepoints_active)`; the begin/commit/rollback hooks. The one-commit + defer and the extend-only termination invariant live entirely inside this unit. +- **Note:** because `cow_alloc` is shared by the data path, the handle-table COW, + and the membership COW, the closures those call sites pass will be rephrased to + borrow `&mut FreemapRecycle` (a single field) instead of five scattered fields — + a net simplification of the disjoint-borrow dance. + +### `FaultInjector` (fault.rs) — `#[cfg(test)]` +- **State:** `fail_next_membership_op`, `fail_next_handle_table_op`, + `fail_next_update_value_write`, `fail_membership_op_after` (the four `Cell`s). +- **Interface:** `should_fail_membership()`, `should_fail_handle_table()`, etc., + consulted only behind `#[cfg(test)]` in the staging/mutate paths. +- The manager carries `#[cfg(test)] fault: FaultInjector`. Production builds have + zero test fields. + +### `CommitProtocol` (commit.rs) — behavior unit +- **Owns:** the `commit_inner` sequence — the I28 pre-drain flush, the + `FreemapRecycle::persist` call, the data fsync, the superblock build + write + + fsync (the strict data-fsync-before-superblock ordering), the + `committed_roots = current_roots` promotion, and the structural-frees promotion. +- **Operates over:** `&mut PageCache`, `&mut Roots` (current/committed), the + `txn_counter`, `superblock_count`, and `&mut FreemapRecycle`. Owns no state. +- The 3-fsync ordering and the I18 allocate-before-merge invariant are preserved + exactly; this unit makes them a single readable sequence rather than a method + buried among 60 others. + +### `StagingTxn` (staging.rs) — behavior unit +- **Owns:** the BUG#2 atomic staging of `allocate_inner` — the forward + (handle-table) and reverse (membership) candidate computation + (`handle_table_insert_candidate`, `membership_insert_candidate`, + `membership_remove_candidate`), `abort_allocate_prepare`, and the infallible + install phase. The "compute-without-install, then install atomically" + discipline (and the bounded-residue-on-abort contract documented in PR #75) + lives here. +- **Operates over:** the cache, the roots, `&mut HandleTable`, `&mut + MembershipIndex`, `&mut FreemapRecycle`, `&mut SlotPacker`, and (test-only) the + `FaultInjector`. Last to extract because it touches the most units. + +### Slimmed `TransactionManager` (mod.rs) +- **Holds:** `cache`, `committed_roots`, `current_roots`, `handle_table`, + `membership_index`, `txn_counter`, `superblock_count`, `active_txn`, + `savepoints`, `txn_freed_pages`, `poisoned`, and the owned units (`packer`, + `freemap`, `#[cfg(test)] fault`). +- **Is:** the public API surface + the thin orchestration (`begin`/`commit`/ + `rollback` delegate the heavy lifting to the units; `read`/`update`/`delete`/ + named-roots live in their concern files but as `impl TransactionManager`). + +## Behavior preservation & testing + +- **The oracle is the existing suite** (~2,281 lines incl. the recycle pins, the + staging tests, `assert_no_reachable_page_is_free`, the recovery/superblock + tests, the spillway/fsync-count integration tests). A pure refactor changes no + test *behavior*; the file-split re-paths them into `transaction/tests.rs` (or + per-concern test files), and the only test-code change is the fault-flag + plumbing moving to `FaultInjector`. +- **Green at every step:** full `cargo test` (not `--lib`) + `cargo clippy + --workspace --all-targets -- -D warnings` + `cargo fmt --check` + the Python + suite must pass before any step is committed. +- **Per-unit surface check:** after each extraction, confirm the unit's public + (crate-visible) surface is its narrow interface — no field leaks beyond it + (clippy + a grep for direct field access from outside the unit's module). + +## Execution order (six green steps, low-risk → high-risk) + +Each step is its own commit, likely its own PR off `main` (per the project's +one-PR-per-unit workflow), merged before the next begins so each subsequent diff +is small and reviewable against a clean base. + +1. **File-split** — pure code movement into `src/transaction/`; struct + fields + + behavior unchanged. Establishes the module; makes every later diff smaller. +2. **FaultInjector** — consolidate the four `#[cfg(test)]` flags into one + test-only struct field. Smallest, test-only. +3. **SlotPacker** — extract the R1 cluster + its savepoint snapshot/restore hooks. +4. **FreemapRecycle** — extract the recycle cluster; rephrase the `cow_alloc` + call sites to borrow the single unit. The hardest data extraction. +5. **CommitProtocol** — extract `commit_inner` onto the now-clean base. +6. **StagingTxn** — extract the prepare/install. Last (most cross-unit). + +## Risks & mitigations + +- **Borrow-checker pushback on owned orchestrators** → the pre-blessed + function-module fallback (decision 4). The interface stays narrow either way. +- **Subtle behavior change in a pure refactor** → the suite is the net; each step + is small and independently green; the highest-risk steps (commit, staging) land + last on an organized base and get adversarial review. +- **Savepoint snapshot coupling** → `Savepoint` captures `live_slots` + + `insert_cursor`; `SlotPacker` exposes `snapshot()/restore()` so the savepoint + machinery snapshots through the narrow interface rather than reaching fields. +- **#75 (abort-leak) in flight** → it edits `transaction.rs`; the file-split must + start off `main` only after #75 merges, or the move-vs-edit conflict is + unmanageable. + +## Format-version / Don't-Break compliance + +Pure structural refactor: no on-disk byte changes meaning, no commit-ordering +change, no poison-model change, no API change, no `FORMAT_VERSION` impact. The +3-fsync ordering, the I18 ordering, the extend-only freemap termination, the +single-writer `&mut self` contract, and strict layering are all preserved — the +refactor only relocates the code that enforces them. + +## Out of scope + +- Any behavior change, optimization, or new feature (this is purely structural). +- Splitting the test suite into a separate crate. +- Extracting reads/mutations/named-roots/stats/config into owned units — they + move to concern files (file-split) but stay `impl TransactionManager`; they are + thin delegators without a distinct state cluster, so a sub-struct would be + ceremony without benefit (YAGNI). +- A `dyn` trait-object fault-injection seam (decision 5). diff --git a/src/transaction.rs b/src/transaction.rs deleted file mode 100644 index 934723b..0000000 --- a/src/transaction.rs +++ /dev/null @@ -1,5580 +0,0 @@ -// transaction.rs — Transaction lifecycle, savepoints, commit protocol, and data operations. -// This is the orchestration layer (layer 6 in the module graph per ARCHITECTURE.md) that ties -// together the handle table, data pages, overflow pages, freemap, superblock, and page -// cache into a coherent transactional API. -// -// Durability model (shadow paging, no WAL): -// - Writes never overwrite live pages. Mutations go to freshly allocated pages via -// PageCache::new_page() and the new roots are threaded through a rebuilt handle -// table spine (COW). The previously-committed pages remain intact on disk until -// the new superblock supersedes them. -// - A commit becomes visible atomically when a new superblock with a higher -// txn_counter and a valid checksum is fsync'd to its (alternating) slot. -// - Crash recovery = open_existing() runs Superblock::select() and picks the -// highest-txn_counter superblock with a valid checksum. A torn/partially-written -// new superblock fails its checksum, so the previous committed state wins — -// no log replay, no undo. -// -// Concurrency model: -// - A TransactionManager is single-writer. active_txn guards against nested begin(). -// Multi-process exclusion is enforced at the file layer by flock() in PageIo; -// only one TransactionManager may hold the database open at a time. -// - TransactionManager is NOT internally thread-safe — callers must serialize -// access. Readers and writers share the same PageCache; there is no MVCC. -// -// In-memory vs on-disk state during an open transaction: -// - All mutations live in the PageCache as dirty entries. Nothing mutated by the -// transaction is durable (or even written to the file in general) until commit(). -// - The superblock on disk still points at committed_roots; current_roots lives -// only in memory. A crash mid-transaction discards all dirty pages from cache -// and the on-disk superblock still references the prior committed snapshot. -// - NOTE: `new_page()` (file extension) extends the underlying file immediately; -// `allocate_data_page` prefers reuse from `current_freemap` but also calls -// through to `new_page()` when the freemap is empty. Either way, any pages -// extended-but-uncommitted before a crash are harmless because nothing in the -// committed superblock references them, and the rollback path -// (`cache.truncate(committed_roots.total_pages)` — I3) actively shrinks the -// file on a clean rollback so they don't accumulate at all. -// -// In-memory mode: `TransactionManager::create_new` and `open_existing` are -// backend-agnostic — whether the underlying PageIo is backed by a file (with -// flock) or by a Vec (no flock, no durability) is invisible here. The -// in-memory entry points live in `lib.rs` and just hand this module a -// memory-backed PageIo. Every transactional invariant in this file (commit -// ordering, poison on fatal error, watermark rollback) applies equally to the -// in-memory backend. - -use std::cell::{Cell, RefCell}; -// I127 (ISSUES.md, 2026-06-21): FxHashMap (not std SipHash) for the per-op -// slot-accounting maps below (current/committed_live_slots, Savepoint.live_slots). -// Keys are trusted local u64 page ids — no DoS surface — so SipHash is pure cost, -// exactly the I77 rationale; that pass converted the page cache/LRU but missed -// these. FxHashMap is a drop-in std HashMap with a faster non-DoS-resistant hasher. -use rustc_hash::{FxHashMap, FxHashSet}; - -use crate::data_page::DataPage; -use crate::error::{ChiselError, Result}; -use crate::freemap_tree::FreeMapTree; -use crate::handle_table::{HandleEntry, HandleFlags, HandleTable}; -use crate::membership_index::{MembershipIndex, RadixU64}; -use crate::overflow::Overflow; -use crate::page::{self, PAGE_ID_NONE, PAGE_SIZE}; -use crate::page_cache::PageCache; -use crate::stats::ChiselCounters; -use crate::superblock::{ - NamedRoot, Superblock, MAX_SUPERBLOCKS, NAMED_ROOT_COUNT, NAMED_ROOT_NAME_LEN, -}; - -// Largest value stored inline in a data-page slot. Larger values are written to an -// overflow chain and referenced by a single HandleEntry with HandleFlags::Overflow. -// -// I117 (ISSUES.md, 2026-06-21): COMPUTED from the page constants (was a -// hand-maintained `8162` literal with only a prose "keep in sync" note). A data -// page's usable body is `CHECKSUM_OFFSET - DATA_PAGE_HEADER_SIZE`, minus one -// `SLOT_ENTRY_SIZE` slot-directory entry. Deriving it makes drift impossible — -// which is what makes the `.expect("value fits in empty page")` in -// `insert_into_data_page` safe by construction: a value `<= MAX_INLINE_VALUE` -// always fits an empty page, so that expect is structurally unreachable. -const MAX_INLINE_VALUE: usize = - page::CHECKSUM_OFFSET - page::DATA_PAGE_HEADER_SIZE - crate::data_page::SLOT_ENTRY_SIZE; - -/// Freemap-aware page allocator shared by data-page allocation and the -/// handle-table / membership-index COW paths. -/// -/// When `reuse_enabled`, it asks the freemap `tree` for the lowest free id -/// at/above `*hint` (clearing its bit via a COW so it cannot be handed out -/// twice), falling back to extending the file via `PageCache::new_page`. -/// `reuse_enabled` is false while savepoints are active (R2: savepoint scopes -/// disable freemap reuse to keep `rollback_to` semantics simple) — matching the -/// historical `allocate_data_page` behavior, which this also routes through. -/// -/// The tree's own COW of the claimed leaf supersedes pages, which the caller -/// drains from `tree.pending_superseded` into `txn_freed_pages` after the call. -/// -/// LAZY-CREATE GUARD: a fresh database has `tree.root == PAGE_ID_NONE` (no tree -/// materialized yet). `PAGE_ID_NONE` is `u64::MAX`, NOT the tree's internal -/// zero-child sentinel, so `allocate_first` would try to read page u64::MAX and -/// error rather than reporting "nothing free". We short-circuit that here: a -/// None-root tree holds nothing reusable, so we fall straight through to -/// `new_page`. The tree is first materialized when a page is *freed* (see -/// persist_freemap), never on the allocation side. -/// -/// Pages freed during the CURRENT transaction live in `txn_freed_pages` and are -/// NOT in the committed tree until commit, so `allocate_first` can never hand -/// back a page still referenced by the live tree (the I18 invariant). Routing -/// handle-table and membership COW allocation through here — rather than the -/// monotonic `new_page` — is what lets those structures reach a bounded -/// steady-state page count instead of leaking one page per mutation. -fn cow_alloc( - cache: &mut PageCache, - tree: &mut FreeMapTree, - hint: &mut u64, - structural_reuse: &mut Vec, - reuse_enabled: bool, -) -> Result { - if reuse_enabled && tree.root != PAGE_ID_NONE { - // `allocate_first` claims a free DATA page (clearing its bit), which COWs - // the freemap leaf. That leaf COW's structural `extend` reuses a dead - // freemap page from `structural_reuse` before extending the file — what - // keeps the freemap from marching the file upward one page per commit. - let mut extend = |c: &mut PageCache| structural_extend(c, structural_reuse); - if let Some(id) = tree.allocate_first(cache, hint, &mut extend)? { - cache.claim_page(id)?; - return Ok(id); - } - } - cache.new_page() -} - -// Verification hook (tests only): every page id drawn from `structural_reuse` -// as a freemap-COW target is recorded here, so the recycle pin-tests can assert -// the one-commit defer (a reused id was promoted by a PRIOR commit, never one -// this transaction itself superseded). A thread-local keeps the production -// `structural_extend` signature and both inline pop sites untouched; the -// recording calls are `#[cfg(test)]` no-ops in release builds. The single-writer -// model means at most one manager drives this per thread at a time. -#[cfg(test)] -thread_local! { - static STRUCTURAL_REUSE_LOG: RefCell> = const { RefCell::new(Vec::new()) }; -} - -#[cfg(test)] -fn record_structural_reuse(id: u64) { - STRUCTURAL_REUSE_LOG.with(|log| log.borrow_mut().push(id)); -} - -/// Drain and return every structural-reuse pop recorded since the last drain. -#[cfg(test)] -fn take_structural_reuse_log() -> Vec { - STRUCTURAL_REUSE_LOG.with(|log| std::mem::take(&mut *log.borrow_mut())) -} - -/// Structural-page allocator for the freemap tree's COW: reuse a dead freemap -/// page (deferred from a prior commit, now safe to overwrite) before extending -/// the file. NEVER draws from the freemap's own free bits — that would re-COW a -/// leaf and recurse — preserving the extend-only termination guarantee while -/// bounding steady-state growth. `claim_page` evicts any stale cache entry for -/// the reused id before the COW writes its fresh contents. -fn structural_extend(cache: &mut PageCache, structural_reuse: &mut Vec) -> Result { - if let Some(id) = structural_reuse.pop() { - #[cfg(test)] - record_structural_reuse(id); - cache.claim_page(id)?; - Ok(id) - } else { - cache.new_page() - } -} - -/// Snapshot of the mutable "pointers" that define a consistent database state. -/// A commit succeeds by writing a superblock that references exactly these roots; -/// a rollback succeeds by reverting current_roots back to committed_roots. -/// -/// The `named_roots` array is part of this snapshot (ISSUES.md F2) so that -/// set_root_name / clear_root_name participate in the transactional commit -/// point for free — a rollback or `rollback_to` restores named roots at -/// the same time it restores the handle-table root, with no extra plumbing. -#[derive(Debug, Clone)] -struct Roots { - handle_table_page: u64, - // Root page of the freemap tree (PageType::FreeMap leaf at depth 0, or a - // FreeMapInterior at depth > 0). PAGE_ID_NONE until the first free - // materializes the tree (see persist_freemap). Paired with `freemap_depth`, - // these two words ARE the committed freemap; cloning Roots at - // begin/commit/rollback/savepoint carries them with no extra plumbing. - freemap_page: u64, - // Depth of the freemap tree rooted at `freemap_page`. Depth 0 = today's - // single-leaf format (a lone FreeMap page reached directly), so existing - // databases load unchanged. Grows logarithmically with database size. - freemap_depth: u32, - next_handle: u64, - total_pages: u64, - named_roots: [NamedRoot; NAMED_ROOT_COUNT], - // Root page of the membership index (chunk-tags). PAGE_ID_NONE until the - // first tagged chunk is written. Cloned automatically with the rest of - // Roots at begin/commit/rollback/savepoint — no extra plumbing needed. - membership_index_page: u64, -} - -/// A nested rollback point within an active transaction. -/// -/// Captures the roots and the `next_page_id` watermark at savepoint -/// creation time. `rollback_to(name)` restores the roots and calls -/// `cache.truncate(watermark)`, which drops every cache entry and -/// truncates the file back to the watermark — cleanly discarding every -/// page the transaction allocated after the savepoint (ISSUES.md I3). -/// -/// `freed_pages` is still tracked per-savepoint so a future freemap -/// reclamation pass (R2) can restore freed-but-not-yet-reclaimed pages -/// if a savepoint is rolled back to. It is a distinct concern from the -/// cache-level rollback that the watermark handles. -/// -/// `live_slots` and `insert_cursor` snapshot the R1 packing state -/// (live slot counts per data page + the current in-progress insert -/// cursor). `rollback_to` restores these so a savepoint rewind leaves -/// the packer in a consistent state. Cloning the HashMap is O(map -/// size) per savepoint but savepoints are rare in the target workloads -/// (drop_table / delete_many don't use them). -#[derive(Debug)] -struct Savepoint { - name: String, - roots: Roots, - watermark: u64, - freed_pages: Vec, - live_slots: FxHashMap, - insert_cursor: Option, -} - -/// The single writer for a Chisel database. Not thread-safe; file-level mutual -/// exclusion across processes is provided by flock() in PageIo. Holds both the -/// last durably-committed roots (for reads outside a txn and for rollback) and -/// the in-progress current_roots (only valid while active_txn is true). -pub struct TransactionManager { - // Interior mutability (ISSUES.md F3): the page cache is mutated on read - // (LRU bookkeeping, page loads, checksum validation), but from Chisel's - // public API perspective a read() is semantically a read. Wrapping in - // RefCell lets `read()` / `handles()` / `stats()` take `&self` so - // callers don't need an external RefCell wrapper. RefCell (not - // Mutex) because Chisel is deliberately single-threaded — see - // lib.rs and ARCHITECTURE.md. Every access through this field uses - // `borrow_mut()`; reborrowing for downstream `&mut PageCache` parameters - // (e.g., handle_table methods) is done via `&mut *cache` on a single - // RefMut held for the duration of the operation. - cache: RefCell, - // Roots that match the superblock currently on disk. Safe to read at any time. - committed_roots: Roots, - // Roots under construction. Equals committed_roots when no txn is active; - // diverges from it as mutations create new COW pages during a txn. - current_roots: Roots, - handle_table: HandleTable, - /// In-memory state for the membership index (chunk tags). Holds only the - /// outer tree's depth; the root lives in current/committed `Roots`. - membership_index: MembershipIndex, - // Monotonically increasing. Written into each new superblock; the higher value - // wins on recovery. Also used to pick the inactive slot on commit via - // `txn_counter % superblock_count`. - txn_counter: u64, - // Number of superblock slots occupying pages 0..superblock_count - // (ISSUES.md R4). Set at open time from the winning superblock's - // own `superblock_count` field; cached here so commit doesn't have - // to re-fetch it. Must equal every slot's self-reported value in a - // healthy database; divergence would indicate mid-flight reconfig - // or corruption. - superblock_count: u32, - active_txn: bool, - savepoints: Vec, - // Pages whose contents are no longer reachable from the new roots. - // Merged into `current_freemap` at commit time so subsequent - // transactions can reuse the space (ISSUES.md I9 / I10 / I11 / R2). - // During the transaction itself these pages are NOT reusable — - // their old contents must stay readable via `committed_roots` until - // commit promotes the new roots. - txn_freed_pages: Vec, - // Best-effort lower bound on the lowest free page id in the committed - // freemap tree, threaded into `FreeMapTree::allocate_first` so a scan - // starts near the answer instead of at id 0. Deliberately NOT - // transactionally tracked: a too-low hint only costs a wasted left-to-right - // scan, never correctness (the scan still returns the true lowest free id), - // so it needs no begin/rollback snapshotting. `allocate_first` advances it; - // a free at a lower id is invisible to the hint until the next scan walks - // back over it, which is acceptable slack. Init 0. - freemap_hint: u64, - // Dead freemap pages carried BETWEEN commits, the engine's bounded-growth - // mechanism for the extend-only freemap (ISSUES.md I18, generalized to the - // tree). Lifecycle: - // - // * A freemap mutation (data-alloc-side leaf COW, or persist's frees) must - // COW the committed freemap pages it touches — it can never overwrite a - // page the last-durable superblock still references. Each COW supersedes - // an OLD freemap page. - // * That old page cannot be reused IN THE SAME COMMIT (the commit's new - // freemap root may still reference it until the superblock flips), so it - // is DEFERRED one commit: collected in `structural_superseded` this - // transaction, promoted to `pending_structural_frees` at commit. - // * The NEXT transaction reuses them: `begin()` moves them into - // `structural_reuse`, and every structural `extend` (freemap COW target) - // pops from that pool before extending the file. This is what makes the - // freemap leaf ROTATE among a small set of pages instead of marching the - // file upward ~1 page/commit forever. Reusing a DEAD page (vs. a free bit - // in the tree) keeps the extend-only TERMINATION guarantee — no freemap - // mutation ever draws structural space from the freemap's own bits. - // - // Not data-reusable (never enters `txn_freed_pages`): a freed freemap page - // sits at a high id, and the lowest-first data allocator would starve it, so - // routing it back as structural reuse (where demand matches supply at steady - // state) is what actually reclaims it. - pending_structural_frees: Vec, - // The dead-freemap-page pool available to reuse as structural COW targets in - // the CURRENT transaction. Seeded from `pending_structural_frees` at - // `begin()`; drained by every structural `extend`; the unconsumed remainder - // is carried forward (back into `pending_structural_frees`) at commit. On - // rollback it is moved back wholesale, restoring the pre-transaction - // `pending_structural_frees`. - structural_reuse: Vec, - // This transaction's freemap-COW supersedes (old freemap pages this txn - // replaced). Accumulated as transient handles drain `tree.pending_superseded` - // here via `put_freemap_tree`; promoted to `pending_structural_frees` at - // commit (the one-commit defer). Dropped on rollback (those COWs are - // truncated above the watermark). - structural_superseded: Vec, - // Freemap pages already COW'd/extended by the CURRENT transaction. Because - // the manager rebuilds a transient `FreeMapTree` handle at every allocation - // site (data-page alloc, each HT/membership COW, persist_freemap), this set - // is what lets those handles share the "first touch this txn => COW, later - // touches => in-place" discipline: without it every site would re-COW the - // same freemap leaf, turning reclamation into unbounded file growth. Swapped - // into each transient handle and read back out (see `freemap_tree` helper). - // Cleared at begin (fresh per transaction); also cleared on commit/rollback - // so the next transaction starts empty. A stale entry pointing at a - // now-committed page would be a CORRECTNESS bug (it would suppress a needed - // COW and mutate a live committed page in place), which is exactly why it is - // transaction-scoped, not cross-transaction. - freemap_session_owned: FxHashSet, - // Live-slot count per data page (ISSUES.md R1). Tracks how many - // handle-table entries currently point at each data page — this - // is the information needed to decide when a page is fully empty - // and can be returned to the freemap. `committed_live_slots` is - // the durable state (rebuilt at open time by scanning the handle - // table); `current_live_slots` is the in-transaction working copy. - // - // Kept in memory rather than on disk because updating a slot count - // on a committed data page would require COW, and COWing a data - // page would require rewriting every handle_table entry that - // points into it — an O(live-slots-in-page) amplification per - // delete that shadow paging does not handle well. - committed_live_slots: FxHashMap, - current_live_slots: FxHashMap, - // Per-transaction "insert cursor" (ISSUES.md R1). The id of a data - // page allocated earlier in the current transaction that still has - // free space. New values pack into it until it fills, at which - // point a new page is allocated and becomes the new cursor. - // - // `None` at the start of each transaction. Only set for pages - // allocated during THIS transaction (so they're dirty in the cache - // and safe to modify). A committed data page is never the cursor — - // that would require COW, which is prohibitively expensive for data - // pages (every handle_table entry pointing at the page would need - // to be rewritten). Disabled entirely when savepoints are active, - // same as freemap reuse (R2): the savepoint-snapshot cost becomes - // manageable when only one code path interacts with packing state. - insert_cursor: Option, - // Poison flag (ISSUES.md I1). Once set, every public entry point returns - // ChiselError::Poisoned until the manager is dropped. Set by commit() on - // any error in the commit protocol, and by `poison_on_fatal()` for any - // fatal error observed during other operations. Modeled on - // std::sync::Mutex poisoning: the only legal recovery is to drop the - // Chisel handle and reopen; the shadow-paging crash-recovery logic then - // returns the database to the last durable state. Linux fsync semantics - // (fsyncgate, 2018) make this the ONLY safe response to a mid-commit - // I/O error — a failed fsync cannot be retried without first closing - // and reopening the file. - // - // Stored as `Cell` (not plain `bool`) so it can be set from the - // `&self`-taking read paths introduced by F3. Cell rather than - // AtomicBool because TransactionManager is !Sync by design (see - // lib.rs); there is no cross-thread access to synchronize against. - poisoned: Cell, - - // Test-only fault injection (BUG#2 atomic-staging regression tests). When - // armed, the NEXT reverse-map (membership-index) update in - // `allocate_inner`/`delete_inner` returns a non-fatal `CacheFull` BEFORE - // touching the index, simulating a mid-operation resource-exhaustion strike - // at exactly the forward/reverse divergence window. Gated `#[cfg(test)]` so - // it carries no production code (mirrors `force_poison_for_test`). Cell so - // the in-crate tests can arm it through `&self`. - #[cfg(test)] - fail_next_membership_op: Cell, - - // Companion to `fail_next_membership_op` for the FORWARD step: when armed, - // the next `allocate_inner` handle-table insert returns a non-fatal - // `CacheFull`, so tests can exercise the prepare-abort/unwind path of the - // step that carries the eager depth bump (HandleTable::grow). `#[cfg(test)]`. - #[cfg(test)] - fail_next_handle_table_op: Cell, - - // For `update_inner`: when armed, the next update returns a non-fatal - // `CacheFull` at the NEW-value-write step. With the fix this is the first - // fallible step (a clean no-op); pre-fix it lands AFTER the old location was - // already freed, so the regression test can prove the old value is not - // prematurely freed before the new entry installs. `#[cfg(test)]`. - #[cfg(test)] - fail_next_update_value_write: Cell, - - // Countdown variant of `fail_next_membership_op` for multi-delete passes - // (e.g. delete_with_tag): when set to K, the Kth subsequent membership-index - // op fails with a non-fatal CacheFull (the first K-1 succeed). Lets a test - // fail a LATER delete in a loop so earlier deletes commit first. 0 disables. - // `#[cfg(test)]`. - #[cfg(test)] - fail_membership_op_after: Cell, -} - -impl TransactionManager { - /// Create a new database with `superblock_count` superblock slots. - /// - /// All N slots are initialized as VALID superblocks at staggered - /// counters 0..N-1 (slot i gets counter N-1-i). This matters for - /// crash safety (ISSUES.md I2 + R4): - /// - /// * The I2 fix for N=2: if the first user commit (which writes - /// slot 0 at counter N) is torn, slot 1 at counter N-2 still - /// holds a valid "empty database" superblock so the file stays - /// openable. - /// * The R4 generalization for N>=3: multiple staggered fallback - /// slots survive CONSECUTIVE torn writes. For N=3, slots 1 and 2 - /// both hold valid empty states at lower counters after slot 0 - /// is written; a torn retry of the same commit still has slot 2 - /// to fall back to. - /// - /// An fsync is issued before returning so the whole bank of slots - /// is durable before any user data is written. Slot counters with - /// the value 0 are SAFE even though zero bits are "the natural - /// value of an uninitialized disk region" because `select()` - /// filters on XXH3 checksum validity BEFORE comparing counters — - /// a legitimate counter-0 slot has a valid checksum; a zeroed - /// region doesn't. - pub fn create_new(mut cache: PageCache, superblock_count: u32) -> Result { - // Caller is expected to have validated bounds via Options in - // lib.rs, but defend against direct-call misuse too. - assert!( - (2..=MAX_SUPERBLOCKS).contains(&superblock_count), - "superblock_count {superblock_count} out of supported range 2..=16" - ); - - // Write N staggered slots. Slot 0 gets the highest counter - // (superblock_count - 1), slot N-1 gets 0. First user commit - // bumps to N, which modulo N is 0, so slot 0 is the first to - // be overwritten — the behavior the I2 fix established for - // N=2 generalizes cleanly to larger N. - // - // Invariant after this loop: every slot is a valid superblock - // referencing the same (empty) roots, at counters 0..N-1. - // `select()` at open time will pick slot 0 (highest counter). - // After the first user commit, slot 0 holds the newest data - // and the rest remain as "rollback fallbacks". - let mut sb = Superblock::new_empty(superblock_count); - for i in 0..superblock_count { - sb.txn_counter = (superblock_count - 1 - i) as u64; - let buf = sb.serialize(); - cache.io_mut().write_page(i as u64, &buf)?; - } - cache.io_mut().fsync()?; - cache.set_next_page_id(superblock_count as u64); - - let roots = Roots { - handle_table_page: PAGE_ID_NONE, - // No freemap tree yet: the first allocation falls through to extend - // (nothing to reuse) and the first free materializes the tree lazily - // (persist_freemap calls FreeMapTree::create). Depth 0 matches the - // single-leaf format. - freemap_page: PAGE_ID_NONE, - freemap_depth: 0, - // Start at 1: handle 0 is reserved as the "no handle" sentinel and is - // never minted (see Superblock::new_empty, which seeds the persisted - // superblock the same way). Must match new_empty so the in-memory - // roots and the on-disk superblock of a fresh store agree. - next_handle: 1, - total_pages: superblock_count as u64, - named_roots: [NamedRoot::EMPTY; NAMED_ROOT_COUNT], - membership_index_page: PAGE_ID_NONE, - }; - - Ok(TransactionManager { - cache: RefCell::new(cache), - committed_roots: roots.clone(), - current_roots: roots, - handle_table: HandleTable::new(), - membership_index: MembershipIndex::new(), - // Slot 0 was written last in the loop above, at counter - // (superblock_count - 1 - 0) = superblock_count - 1. That's - // the highest counter and therefore the winner on select(). - txn_counter: (superblock_count - 1) as u64, - superblock_count, - active_txn: false, - savepoints: Vec::new(), - txn_freed_pages: Vec::new(), - freemap_hint: 0, - pending_structural_frees: Vec::new(), - structural_reuse: Vec::new(), - structural_superseded: Vec::new(), - freemap_session_owned: FxHashSet::default(), - // A fresh database has no data pages and no live slots yet. - committed_live_slots: FxHashMap::default(), - current_live_slots: FxHashMap::default(), - insert_cursor: None, - poisoned: Cell::new(false), - #[cfg(test)] - fail_next_membership_op: Cell::new(false), - #[cfg(test)] - fail_next_handle_table_op: Cell::new(false), - #[cfg(test)] - fail_next_update_value_write: Cell::new(false), - #[cfg(test)] - fail_membership_op_after: Cell::new(0), - }) - } - - /// Open an existing database from file. - /// - /// This is the crash recovery path. All N superblock slots (where - /// N is discovered from disk — see the probe below) are read, - /// `Superblock::select()` picks the one with the highest - /// txn_counter and a valid XXH3 checksum, and a torn write to the - /// most-recently-targeted slot silently falls back to the next - /// best survivor. No log replay required. - /// - /// R4 slot discovery: - /// - /// The number of superblock slots is NOT a compile-time constant. - /// A database created with `superblock_count=4` has 4 slots at - /// pages 0..3; a default database has 2 at pages 0..1. To find N - /// without any external hint we: - /// - /// 1. Read the first MAX_SUPERBLOCKS pages of the file (bounded - /// by EOF — a fresh DB has exactly N pages and no more). We - /// deliberately do NOT short-circuit on "this page doesn't - /// look like a superblock": a torn write that hit the magic - /// bytes of an otherwise-valid slot would look like garbage, - /// and short-circuiting would skip past the legitimate - /// successor slots. Reading a few extra pages is cheap. - /// 2. Pass all candidates to `Superblock::select`, which uses - /// `deserialize` to filter on XXH3 checksum + MAGIC bytes. - /// Data pages that happen to sit at positions < MAX_SUPERBLOCKS - /// (e.g., in a database where N=2 and there's a data page at - /// page 2) fail the magic check and are harmlessly ignored. - /// 3. The winner's `superblock_count` field tells us N, which we - /// cache on the TransactionManager for commit-time slot - /// selection. - /// - /// If no valid superblock is found in the first MAX_SUPERBLOCKS - /// pages, we return `CorruptSuperblock`. This bounds the probe - /// cost in the pathological case where every candidate is torn. - pub fn open_existing(mut cache: PageCache) -> Result { - // Step 1: read up to MAX_SUPERBLOCKS pages as candidates. - let mut candidates: Vec<[u8; PAGE_SIZE]> = Vec::new(); - for i in 0..MAX_SUPERBLOCKS as u64 { - // If the file is shorter than MAX_SUPERBLOCKS (fresh DB - // with small N), read_page returns InvalidPageId (I16). - // Stop probing at EOF. - match cache.io_mut().read_page(i) { - Ok(buf) => candidates.push(buf), - Err(ChiselError::InvalidPageId { .. }) => break, - Err(e) => return Err(e), - } - } - - // Step 2 + 3: pick the winner via select(). select() uses - // deserialize, which validates checksum and magic — data - // pages in the candidate list (if any) are filtered out. - let sb = Superblock::select(&candidates).ok_or_else(|| ChiselError::CorruptSuperblock { - defects: Superblock::diagnose(&candidates), - })?; - - // Format-version gate (see ISSUES.md I15 for the original check, - // I29 for the major/minor split). Compare MAJOR only: the packed - // u32 layout (upper 16 = major, lower 16 = minor) lets same-major - // files open regardless of minor drift, which is what makes the - // README's "sacred within a major version" promise enforceable. - // Minor-newer files are accepted (read-compatible) here but forced - // read-only by the I29 write-gate immediately below — writing them - // would drop fields this binary doesn't know about. - // - // We validate AFTER select() rather than inside deserialize() - // because the winning superblock's version is what determines - // compatibility — silently falling back to an older-version - // superblock would hand the user a stale snapshot with - // mysteriously missing data. - if page::format_major(sb.format_version) != page::FORMAT_MAJOR_VERSION { - return Err(ChiselError::UnsupportedFormatVersion { - found: sb.format_version, - expected: page::FORMAT_VERSION, - }); - } - - // Reject files written with a different page geometry before reading any - // data pages — every page boundary calculation would be wrong if the page - // size differed. The superblock.rs deserialize() reads the field but does - // not validate it (a size mismatch is not a torn-slot signal; it must not - // cause select() to fall back to a sibling slot). This is the right place - // to raise it: after select() has picked the winning slot but before any - // data is touched. - if sb.page_size != PAGE_SIZE as u32 { - return Err(ChiselError::UnsupportedPageSize { - stored: sb.page_size, - compiled: PAGE_SIZE as u32, - }); - } - - // I29 write-gate: a file whose MINOR exceeds this binary's may contain - // version-requiring page layouts we cannot safely write — we would - // stamp pages at our older minor and drop the newer fields. Reads ARE - // safe (within a MAJOR all layout changes are additive, so known fields - // sit at stable offsets), so we open the file but force it read-only; - // mutations then return ReadOnlyMode. The complementary I31 per-page - // read-dispatch lets a newer binary read these older pages. - // See docs/specs/2026-06-21-per-page-format-versioning-design.md. - if page::format_minor(sb.format_version) > page::FORMAT_MINOR_VERSION { - cache.io_mut().force_read_only(); - } - - let page_count = cache.io_mut().page_count()?; - if page_count < sb.total_pages { - return Err(ChiselError::FileSizeMismatch { - // saturating_mul: `sb.total_pages` comes from a checksum-valid but - // otherwise untrusted superblock — `Superblock::deserialize` bounds - // only the checksum, MAGIC, and superblock_count, NOT total_pages. - // A crafted/edited file with total_pages near u64::MAX would - // overflow `* PAGE_SIZE` here: a panic in debug builds (how CI - // runs) and a silent wrap in release. Saturating keeps the public - // `Chisel::open` a typed-error path; "as many bytes as a u64 can - // represent" is the right report for an absurd page count (mirrors - // the I47 saturation in `Chisel::stats`/`file_size_bytes`). - // `page_count` is file-length-bounded and cannot realistically - // overflow, but it is saturated too for symmetry. - expected: sb.total_pages.saturating_mul(PAGE_SIZE as u64), - actual: page_count.saturating_mul(PAGE_SIZE as u64), - }); - } - // Reset next_page_id from the authoritative superblock, NOT from - // the on-disk file length (ISSUES.md I4). This matters because a - // crash mid-rollback could leave the file extended past the - // committed superblock's `total_pages` — those trailing pages are - // unreferenced garbage, and letting `new_page()` allocate above - // them would mean the next commit's new pages live at the very - // end of the file while the garbage sits in the middle. Reseeding - // from `sb.total_pages` causes the next allocations to overwrite - // the garbage, which is exactly what we want. The rollback-path - // truncation added by I3 also prevents this situation from - // arising in the first place, but the reseed is a defense-in- - // depth guarantee against any crash that happened before I3 or - // against external truncation/corruption tools. - cache.set_next_page_id(sb.total_pages); - - let roots = Roots { - handle_table_page: sb.root_handle_table_page, - // The committed freemap tree IS {root, depth} from the superblock — - // no separate in-memory mirror is loaded. Depth 0 (the default for - // pre-multi-page databases) reaches today's single-leaf format. - freemap_page: sb.root_freemap_page, - freemap_depth: sb.freemap_depth, - next_handle: sb.next_handle, - total_pages: sb.total_pages, - named_roots: sb.named_roots, - // Normalize old files (pre-chunk-tags bytes were zeroed) so the - // rest of the engine has a single "empty" sentinel: PAGE_ID_NONE. - membership_index_page: if sb.root_membership_index_page == 0 { - PAGE_ID_NONE - } else { - sb.root_membership_index_page - }, - }; - - // The HandleTable struct keeps only its depth in memory; physical pages - // live in the cache, reached via the root page_id in the superblock. - // Reconstruct depth by walking the left spine (see HandleTable::recover_depth). - let mut ht = HandleTable::new(); - ht.set_depth(HandleTable::recover_depth( - &mut cache, - sb.root_handle_table_page, - )?); - - // Mirror the handle-table depth recovery for the membership index: its - // outer RadixU64 keeps only depth in memory, rebuilt by walking the - // persisted spine from the root recorded in the superblock. Uses the - // normalized roots.membership_index_page (PAGE_ID_NONE for legacy files). - let mut membership_index = MembershipIndex::new(); - if roots.membership_index_page != PAGE_ID_NONE { - let depth = RadixU64::recover_depth(&mut cache, roots.membership_index_page)?; - membership_index.set_outer_depth(depth); - } - - // The committed freemap tree is reconstructed on demand from - // {root_freemap_page, freemap_depth} via FreeMapTree::from_roots — no - // eager in-memory mirror is loaded here. A DB created under v1 (pre-R2) - // or a fresh one has root_freemap_page == PAGE_ID_NONE, which - // from_roots treats as the empty (nothing-free) tree. Tree pages are - // checksum-validated on cache miss as they are descended, so a torn or - // corrupt freemap surfaces as a fatal error rather than silent reuse. - - // Rebuild the live-slot count map (ISSUES.md R1) by scanning the - // handle table. Every Live entry contributes one live slot to - // its target data page; Overflow and Deleted entries don't - // count. Cost is O(live handles), paid once at open. In-memory - // only — the alternative (storing the count on the data page - // itself) would require COWing pages on every delete, which - // shadow paging cannot afford. - let mut committed_live_slots: FxHashMap = FxHashMap::default(); - if sb.root_handle_table_page != PAGE_ID_NONE { - let entries = ht.iter_live(&mut cache, sb.root_handle_table_page)?; - for (_, entry) in entries { - if entry.flags == HandleFlags::Live { - *committed_live_slots.entry(entry.page_id).or_insert(0) += 1; - } - } - } - let current_live_slots = committed_live_slots.clone(); - - Ok(TransactionManager { - cache: RefCell::new(cache), - committed_roots: roots.clone(), - current_roots: roots, - handle_table: ht, - membership_index, - txn_counter: sb.txn_counter, - // R4: discovered from the winning superblock's own - // `superblock_count` field. Cached so commit doesn't have - // to re-look it up for slot selection. - superblock_count: sb.superblock_count, - active_txn: false, - savepoints: Vec::new(), - txn_freed_pages: Vec::new(), - freemap_hint: 0, - pending_structural_frees: Vec::new(), - structural_reuse: Vec::new(), - structural_superseded: Vec::new(), - freemap_session_owned: FxHashSet::default(), - committed_live_slots, - current_live_slots, - insert_cursor: None, - poisoned: Cell::new(false), - #[cfg(test)] - fail_next_membership_op: Cell::new(false), - #[cfg(test)] - fail_next_handle_table_op: Cell::new(false), - #[cfg(test)] - fail_next_update_value_write: Cell::new(false), - #[cfg(test)] - fail_membership_op_after: Cell::new(0), - }) - } - - // --- Poison machinery (ISSUES.md I1) --- - // - // Every public entry point below follows the same wrapper pattern: - // - // pub fn foo(&mut self, ...) -> Result { - // self.check_alive()?; // fast path: refuse if already poisoned - // let result = self.foo_inner(...); - // self.poison_on_fatal(result) // poison iff the inner call returned a fatal error - // } - // - // commit() is the one exception: ANY error from the commit protocol - // poisons (not just fatal variants), because partial-commit state is - // fragile enough that we do not trust the in-memory view after a - // half-finished commit even if the variant would otherwise be - // operational. See commit() for the full reasoning. - - /// Returns Err(Poisoned) if the manager has previously seen a fatal - /// error. Called at the top of every public entry point. Cheap. - /// - /// Takes `&self` because the poison flag lives in a `Cell` - /// (F3: `read()` takes `&self`, and read paths must also check/set - /// the flag). - fn check_alive(&self) -> Result<()> { - if self.poisoned.get() { - return Err(ChiselError::Poisoned); - } - Ok(()) - } - - /// Inspect a Result and set the poison flag if it contains a fatal - /// error. Returns the Result unchanged so the caller can `?` or return - /// it. Never fires on an Ok or on an operational error. - /// - /// Takes `&self` (not `&mut self`) because the flag is a `Cell` — - /// essential for the `&self`-taking read paths under F3. - fn poison_on_fatal(&self, result: Result) -> Result { - if let Err(ref e) = result { - if e.is_fatal() { - self.poisoned.set(true); - } - } - result - } - - /// Force the manager into the poisoned state. Test-only hook used by - /// the I1 regression test to avoid needing a real I/O failure injection. - #[cfg(test)] - pub fn force_poison_for_test(&self) { - self.poisoned.set(true); - } - - /// Test-only: forge a freemap orphan exactly as a crash would leave one. - /// Extend a fresh page, stamp it as a checksum-valid `FreeMapInterior`, and - /// return its id WITHOUT referencing it from the live tree or marking it free - /// — the precise state of a structural-recycle-pool page stranded when an - /// in-memory pool is lost to a crash. The orphan sweep - /// (`reclaim_freemap_orphans`) must reclaim it. Returns the forged page id. - /// - /// FreeMapInterior (not FreeMap) is used deliberately: it cannot be mistaken - /// for a freed-bit leaf, and it exercises the interior arm of the type test. - #[cfg(test)] - pub(crate) fn test_forge_freemap_orphan(&mut self) -> Result { - let mut cache = self.cache.borrow_mut(); - let id = cache.new_page()?; - let buf = cache.get_mut(id)?; - buf.fill(0); - buf[0] = crate::page::PageType::FreeMapInterior as u8; - buf[1] = page::current_version(crate::page::PageType::FreeMapInterior); - page::stamp_checksum(buf); - Ok(id) - } - - /// Test-only: forge a CORRUPT, non-reachable page on disk. Extend a fresh - /// page, fill it with garbage, deliberately do NOT stamp a valid checksum, - /// flush it to the backing file, then drop it from the cache so a later - /// `get(id)` re-reads it from disk and fails with `ChecksumMismatch`. The - /// page is never referenced from any tree, so it is a corrupt DEAD page — - /// exactly what the orphan sweep must SKIP rather than poison on. Returns the - /// forged page id. - #[cfg(test)] - pub(crate) fn test_forge_corrupt_dead_page(&mut self) -> Result { - let mut cache = self.cache.borrow_mut(); - let id = cache.new_page()?; - let buf = cache.get_mut(id)?; - // Garbage bytes with a freemap-ish type byte but a checksum that will not - // verify (we never call stamp_checksum). The type byte is irrelevant — - // the read fails the checksum gate before the type is ever inspected. - buf.fill(0xAB); - buf[0] = crate::page::PageType::FreeMap as u8; - cache.flush()?; // write the garbage bytes to the main file - cache.test_drop_from_cache(id); // force a disk re-read (and checksum check) next get - Ok(id) - } - - /// True if this manager has been poisoned by a previous fatal error. - pub fn is_poisoned(&self) -> bool { - self.poisoned.get() - } - - // --- Watermark-based rollback (ISSUES.md I3 + I7) --- - // - // `PageCache::new_page()` hands out monotonically increasing ids, so - // every page allocated during a transaction has an id strictly greater - // than or equal to the `next_page_id` watermark captured at begin() / - // savepoint() time. `PageCache::truncate(watermark)` drops every cache - // entry AND truncates the file to `watermark` pages, cleanly discarding - // every transaction-allocated page without a per-page tracking list. - // - // This supersedes an earlier per-page `txn_dirty_pages` vector — the - // list was a weaker mechanism (I7 showed it missed intermediate COW - // pages and overflow allocations) and a redundant one once the - // watermark invariant was in place. See memory - // project_chisel_i3_watermark_rollback for the reasoning. - // - // Savepoints capture `cache.next_page_id()` at creation time (see the - // `watermark` field on Savepoint) so `rollback_to(name)` can truncate - // to that specific watermark — discarding every page allocated after - // the savepoint while preserving those allocated before it. - - /// Snapshot the current `next_page_id` watermark. Cheap — one read - /// through the RefCell. - fn cache_watermark(&self) -> u64 { - self.cache.borrow().next_page_id() - } - - // --- Freemap-aware page allocation (ISSUES.md R2) --- - // - // `allocate_data_page` is the single entry point for allocating a - // fresh data page during a transaction. It first tries to reuse an - // id from `current_freemap` and falls back to extending the file. - // - // Two important scoping rules: - // - // 1. Reuse is disabled when any savepoint is active. A rollback_to - // would need to per-savepoint distinguish dirty entries at - // reused ids from dirty entries at preserved ids, which would - // require an 8 KB freemap snapshot per savepoint and a - // per-savepoint dirty-page list. For v1, the simpler rule is - // "reuse only outside savepoint scopes". Workloads that want - // reuse (e.g. F1 delete_subtree / drop_table) typically don't - // use savepoints at all. - // - // 2. Pages freed during the CURRENT transaction (in - // `txn_freed_pages`) are NOT reusable within the same - // transaction — their old contents must stay readable via - // `committed_roots` until commit swaps the superblock. This - // is enforced by only merging `txn_freed_pages` into - // `current_freemap` during commit, after the new roots have - // been computed. - // - // Handle-table and membership-index COW pages now share this same - // freemap-aware allocator via `cow_alloc` (each `insert`/`delete` takes an - // `alloc` closure that calls it), so they reuse freed pages before - // extending — that is what bounds their steady-state page count. Overflow - // pages still call `cache.new_page()` directly and always extend, but their - // frees feed the freemap, so a later data- or handle-table allocation can - // reclaim them. Routing overflow through the freemap too would need the - // same allocator-closure plumbing at the overflow module boundary; left as - // a v1 simplification since overflow churn is far smaller than HT churn. - /// Build a transient `FreeMapTree` handle from the current freemap roots, - /// MOVING the transaction's `freemap_session_owned` set into it so this - /// handle treats pages an earlier site already COW'd this transaction as - /// in-place-mutable. Pair with `put_freemap_tree`, which moves the (possibly - /// grown) set back out — never drop a handle from `take_` without a matching - /// `put_`, or the session set is lost and later sites re-COW. - fn take_freemap_tree(&mut self) -> FreeMapTree { - let mut tree = FreeMapTree::from_roots( - self.current_roots.freemap_page, - self.current_roots.freemap_depth, - ); - tree.session_owned = std::mem::take(&mut self.freemap_session_owned); - tree - } - - /// Write a transient handle's grown root/depth back into the current roots, - /// move its session-owned set back into the manager, and drain its - /// COW-superseded freemap pages into `structural_superseded` (the one-commit - /// defer stream — NOT `txn_freed_pages`, since freed freemap pages are - /// recycled as structural reuse, not as data frees). - fn put_freemap_tree(&mut self, mut tree: FreeMapTree) { - self.current_roots.freemap_page = tree.root; - self.current_roots.freemap_depth = tree.depth; - self.structural_superseded - .append(&mut tree.pending_superseded); - self.freemap_session_owned = std::mem::take(&mut tree.session_owned); - } - - fn allocate_data_page(&mut self) -> Result { - let reuse = self.savepoints.is_empty(); - let mut tree = self.take_freemap_tree(); - let id = { - let mut cache = self.cache.borrow_mut(); - cow_alloc( - &mut cache, - &mut tree, - &mut self.freemap_hint, - &mut self.structural_reuse, - reuse, - ) - }; - // Write back tree growth + drain supersedes even on error: the freemap - // pages were extended (never freed), so on a non-fatal failure they are - // harmless above-watermark scratch, and the session set must still be - // returned so a retry/commit in the same transaction stays consistent. - self.put_freemap_tree(tree); - id - } - - /// COW `handle`'s handle-table entry to `entry`, installing the new root - /// and queuing the superseded spine pages for freemap reclamation at - /// commit. Shared by `allocate`, `update`, and `set_client_byte`. - /// - /// The superseded pages are appended to `txn_freed_pages` ONLY after the - /// new root is installed in `current_roots`: if the COW fails partway - /// (e.g. `CacheFull`), the local `freed` list is dropped and the still- - /// current old tree keeps all its pages — never freeing a live page. - fn ht_insert(&mut self, handle: u64, entry: &HandleEntry) -> Result<()> { - let mut freed: Vec = Vec::new(); - let reuse = self.savepoints.is_empty(); - // Build the freemap-tree handle (with the session set moved in) and - // borrow the hint + structural-reuse pool as locals, all disjoint from - // `self.handle_table`, so the alloc closure (which mutates them) and the - // handle-table insert can both borrow `self` at once. - let mut tree = self.take_freemap_tree(); - let result = { - let hint = &mut self.freemap_hint; - let pool = &mut self.structural_reuse; - let mut cache = self.cache.borrow_mut(); - let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); - self.handle_table.insert( - &mut cache, - self.current_roots.handle_table_page, - handle, - entry, - &mut alloc, - &mut freed, - ) - }; - // Write back freemap growth (its supersedes go to structural_superseded - // via put_freemap_tree). Done before the `?` so a freemap COW that - // happened before an insert error still returns the session set and - // records the extended root. Handle-table supersedes (`freed`) only land - // in txn_freed_pages after the new root is installed. - self.put_freemap_tree(tree); - let new_root = result?; - self.current_roots.handle_table_page = new_root; - self.txn_freed_pages.append(&mut freed); - Ok(()) - } - - // Persist the freemap tree at commit time (ISSUES.md R2 / I11 / I18, - // generalized to the multi-page COW tree). - // - // Called once at the very start of `commit_inner`, BEFORE cache.flush(), so - // the freemap pages it COWs join the same durable write set as every other - // dirty page this transaction produced. - // - // TWO FREE-STREAMS (the load-bearing distinction the reviewer scrutinizes): - // - // * `txn_freed_pages` (DATA frees) — pages freed by this commit's - // data/handle-table/membership COW supersedes. Recorded as FREE in this - // commit's new freemap tree, so the NEXT transaction's data/HT - // allocations can reuse them. Safe to mark now: the new tree becomes - // authoritative only when this commit's superblock flips, by which point - // these pages are genuinely dead. - // - // * `structural_superseded` / `pending_structural_frees` / - // `structural_reuse` (FREEMAP-page frees) — the freemap tree's OWN COW - // supersedes. These are NOT marked free in the tree: a freemap page sits - // at a high id where the lowest-first data allocator would starve it, and - // marking a freemap page free inside the tree that is recording frees - // could cascade. Instead they ride a separate recycle: superseded this - // commit (`structural_superseded`) -> deferred one commit - // (`pending_structural_frees`, since the old page is still referenced - // until the superblock flips) -> reused as structural COW targets next - // transaction (`structural_reuse`). This makes the freemap pages ROTATE - // among a small set rather than marching the file upward ~1/commit. - // - // I18 ORDERING preserved by construction. The structural COW never draws a - // page from the freemap's own free bits (that would re-COW a leaf and - // recurse); it only ever extends the file or reuses a DEAD page from a prior - // commit (one no durable superblock still references). So a to-be-freed id - // can never be handed back to record these same frees — the I18 window - // cannot open. `persist_freemap_does_not_reuse_committed_live_pages` is the - // guardrail. - // - // DEPTH-0 EQUIVALENCE. With one leaf this reduces to: COW the leaf once - // (reusing the prior commit's dead leaf id when available, else extend), set - // the freed bits, defer the old leaf to the structural recycle. Steady-state - // page count matches the pre-tree single-page freemap. - /// Mark a single page id free in the working freemap tree, routing every - /// structural COW target through the pooled `structural_extend` (reuse a dead - /// freemap page before extending the file) and lazily materializing the - /// depth-0 root on first use. Lowers `freemap_hint` to cover `id` so the next - /// `allocate_first` scan can reach it. - /// - /// The ONE marking path shared by `persist_freemap` (this commit's data - /// frees) and `reclaim_freemap_orphans` (the defrag orphan-sweep). Both must - /// flow through the same COW + recycle discipline so the structural reuse pool - /// and supersede streams stay consistent; a second marking implementation - /// could silently diverge from the one-commit-defer crash-safety the recycle - /// depends on. Take/put the tree per call: the session-owned set and the - /// reuse pool persist on the manager across calls, so a multi-id loop still - /// COWs each leaf at most once (the session dedup carries across handles). - fn freemap_mark_free_committed_path(&mut self, id: u64) -> Result<()> { - // Take the working handle WITH the transaction's session set so a leaf an - // earlier call (or this commit's data allocations) already COW'd is - // recognized as in-place here, not re-COW'd. - let mut tree = self.take_freemap_tree(); - // RefCell so the structural-`extend` closure can drain the shared reuse - // pool by `&mut` while the rest of the method still owns `self`. - let structural_reuse = std::cell::RefCell::new(std::mem::take(&mut self.structural_reuse)); - let result = (|| { - let mut cache = self.cache.borrow_mut(); - let mut extend = - |c: &mut PageCache| structural_extend(c, &mut structural_reuse.borrow_mut()); - - // Lazy materialization: a database that has never freed a page has no - // tree yet (root == PAGE_ID_NONE). Create the depth-0 leaf now, before - // marking, since `mark_free_growing` needs a real root to COW. - // Preserve the session set across the swap. - if tree.root == PAGE_ID_NONE { - let session = std::mem::take(&mut tree.session_owned); - tree = FreeMapTree::create(&mut cache, &mut extend)?; - tree.session_owned.extend(session); - } - tree.mark_free_growing(&mut cache, id, &mut extend) - })(); - // Pull the hint back to cover `id`: the hint advances monotonically via - // `allocate_first`, so a too-high hint would start the next scan above - // `id` and never reuse it. A too-low hint only costs a wasted scan. - // (Mirrors the oracle proptest's `hint = hint.min(id)`.) - self.freemap_hint = self.freemap_hint.min(id); - // Return the (partly drained) reuse pool and write the tree back even on - // error: its COW supersedes flow to `structural_superseded` via - // put_freemap_tree; commit promotes structural_superseded + the leftover - // reuse pool into pending_structural_frees (the one-commit defer). - self.structural_reuse = structural_reuse.into_inner(); - self.put_freemap_tree(tree); - result - } - - fn persist_freemap(&mut self) -> Result<()> { - // Nothing freed this commit => the committed tree is still exactly right, - // no COW needed. (Structural reuse / supersede streams are only ever - // non-empty when there were frees, so this single check suffices.) - if self.txn_freed_pages.is_empty() { - return Ok(()); - } - - // Mark this commit's DATA frees free in the new tree via the shared - // marking path. Each call take/puts the tree, but the session-owned set - // persists on the manager, so a leaf hit by several frees is COW'd once. - let freed: Vec = std::mem::take(&mut self.txn_freed_pages); - for id in freed.iter().copied() { - self.freemap_mark_free_committed_path(id)?; - } - self.txn_freed_pages = freed; - Ok(()) - } - - /// Reclaim freemap-typed pages orphaned by a crash that lost the in-memory - /// recycle pool. The structural recycle (decision 6 of the design) is held - /// only in memory, so a crash strands its entries: `FreeMap`/`FreeMapInterior` - /// pages that are no longer reachable from the committed tree and were never - /// marked free in the bitmap (a bounded handful — the last commit's structural - /// supersedes). This sweep walks the live tree to find the reachable set, - /// scans the file for freemap-typed pages that are neither reachable nor - /// already free, and marks each free — routing the mark through the SAME - /// `freemap_mark_free_committed_path` the commit uses (COW + recycle), so a - /// reclaimed orphan lands in the BITMAP (data-reusable), disjoint from the - /// in-memory recycle pool. Requires an active transaction (called by defrag). - /// Returns the count reclaimed. - /// - /// THE EXCLUSION SET (get this exactly right): a page in the CURRENT - /// in-memory recycle pool (`structural_reuse` ∪ `structural_superseded` ∪ - /// `pending_structural_frees`) is LIVE recycling state, NOT an orphan — - /// reclaiming it into the bitmap while it is also pool-reusable would - /// double-hand-out the page. After a crash the pool is empty, so the - /// crash-orphaned pages are correctly flagged; in a normal (no-crash) defrag - /// the live pool is excluded so the two reclamation channels never overlap. - /// - /// Reading each non-reachable page through the cache checksum-verifies it. - /// A page that fails because it is GARBAGE/corrupt (`CorruptPage` / - /// `ChecksumMismatch`) is SKIPPED, not propagated (2026-06-22 review: - /// "skip unreadable dead pages") — a non-reachable page we cannot read - /// cannot be confirmed as a freemap orphan, and a dead page's corruption is - /// irrelevant to correctness. Any OTHER read error (e.g. `IoError`, a real - /// device fault) is propagated and poisons, preserving fail-closed for true - /// hardware faults. The LIVE-tree walk (`reachable_pages`) still propagates - /// fatal on a corrupt LIVE node — only the dead-page scan is softened. The - /// scan is O(total_pages) I/O — off the hot path (defrag), bounded, and - /// acceptable. - pub(crate) fn reclaim_freemap_orphans(&mut self) -> Result { - // Skip the sweep entirely while a savepoint is active. The sweep is the - // ONLY path that COWs the freemap (draining committed-LIVE pages into the - // structural recycle streams) while a savepoint is open — ordinary - // allocation already disables structural reuse under a savepoint - // (`reuse = self.savepoints.is_empty()`). But `rollback_to` rewinds only - // the roots + cache watermark, NOT the structural streams: a page the - // sweep drained into `structural_superseded` would survive the rollback, - // get promoted at commit, and be reused as a COW target in the next - // transaction while the last-durable superblock still references it — - // silent durable freemap corruption. Deferring orphan reclamation to a - // defrag run with no active savepoint avoids the whole interaction, so - // `rollback_to_inner` correctly needs no structural-stream reset. - if !self.savepoints.is_empty() { - return Ok(0); - } - let root = self.current_roots.freemap_page; - let depth = self.current_roots.freemap_depth; - if root == PAGE_ID_NONE { - return Ok(0); // no tree yet => no freemap pages can be orphaned - } - - // Pages that are NOT orphans even though unreachable + not-free: the live - // recycle pool (all three streams). See "THE EXCLUSION SET" above. - let mut excluded: FxHashSet = FxHashSet::default(); - excluded.extend(self.structural_reuse.iter().copied()); - excluded.extend(self.structural_superseded.iter().copied()); - // Belt-and-suspenders: `begin()` clones `pending_structural_frees` - // into `structural_reuse`, so every id here is already covered by the - // `structural_reuse` term above. Kept explicitly so the exclusion - // remains correct if `begin()`'s seeding ever changes. - excluded.extend(self.pending_structural_frees.iter().copied()); - - // Collect orphan ids read-only inside a single cache-borrow scope, then - // drop the borrow before marking (the mark path re-borrows the cache). - let tree = FreeMapTree::from_roots(root, depth); - let mut orphans: Vec = Vec::new(); - { - let mut cache = self.cache.borrow_mut(); - // Upper bound: the allocation high-water (`next_page_id`), NOT the - // committed `total_pages`. After a real crash + reopen these are - // equal (open seeds next_page_id from the committed superblock), and - // every orphan — a structural supersede from a committed transaction — - // sits below it. Using next_page_id also covers a page extended - // earlier in THIS session (e.g. the forge-orphan test), which a stale - // committed total_pages would miss. - let total = cache.next_page_id(); - let reachable = tree.reachable_pages(&mut cache)?; - // Pages 0..superblock_count are superblocks; start the scan above them. - for id in self.superblock_count as u64..total { - if reachable.contains(&id) || excluded.contains(&id) { - continue; - } - // Skip a non-reachable page that is GARBAGE/corrupt rather than - // letting it poison the whole maintenance pass (2026-06-22 review - // decision: "skip unreadable dead pages"). A page that is not in - // the live tree cannot be confirmed as a freemap orphan if we - // cannot read its type, and a DEAD page's corruption does not - // affect correctness — so on `CorruptPage`/`ChecksumMismatch` we - // `continue`. We deliberately PROPAGATE every other read error - // (e.g. `IoError`): a real device fault should still surface and - // poison, not be silently swallowed. NOTE: the live-tree walk - // (`reachable_pages` above) still propagates fatal on a corrupt - // LIVE page — only the dead-page scan is softened. - let buf = match cache.get(id) { - Ok(buf) => buf, - Err(ChiselError::CorruptPage { .. } | ChiselError::ChecksumMismatch { .. }) => { - continue; - } - Err(e) => return Err(e), - }; - let ty = buf[0]; - if (ty == crate::page::PageType::FreeMap as u8 - || ty == crate::page::PageType::FreeMapInterior as u8) - && !tree.is_free(&mut cache, id)? - { - orphans.push(id); - } - } - } - // Mark each orphan free through the shared committed-marking path (COW + - // recycle), landing them in the bitmap as data-reusable space. - for id in &orphans { - self.freemap_mark_free_committed_path(*id)?; - } - Ok(orphans.len() as u64) - } - - /// Begin a new transaction. - /// - /// Single-writer: returns TransactionAlreadyActive if one is already in - /// flight. current_roots is reseeded from committed_roots so that any prior - /// (aborted) in-progress state is discarded. The dirty/freed bookkeeping is - /// cleared — this is the only place (besides commit/rollback) those vectors - /// are zeroed, so callers must not rely on them surviving a begin(). - pub fn begin(&mut self) -> Result<()> { - self.check_alive()?; - let result = self.begin_inner(); - self.poison_on_fatal(result) - } - - fn begin_inner(&mut self) -> Result<()> { - // Fail fast on read-only mounts so callers don't build up - // transaction state only to hit a ReadOnlyMode at the first - // write_page call during commit. - if self.cache.borrow().io().is_read_only() { - return Err(ChiselError::ReadOnlyMode); - } - if self.active_txn { - return Err(ChiselError::TransactionAlreadyActive); - } - self.current_roots = self.committed_roots.clone(); - // The freemap root+depth ride in current_roots (cloned just above), so - // there is no separate freemap working copy to reset here. The hint is - // untracked (a stale hint only costs a scan), so it is left as-is too. - // The session-owned set is strictly per-transaction: a page COW'd last - // transaction is now committed and must NOT be mutated in place, so start - // empty. (begin already requires no active txn, so it is normally empty, - // but clear defensively.) - self.freemap_session_owned.clear(); - // Seed the structural reuse pool from the prior commit's deferred dead - // freemap pages: those superblock-unreferenced pages are now safe to - // reuse as this transaction's freemap COW targets, so the freemap rotates - // among a bounded set instead of extending. CLONE (not move) so - // `pending_structural_frees` stays intact as the rollback fallback — a - // rolled-back transaction never reached commit, so its structural recycle - // is exactly the pre-transaction one. `commit_inner` overwrites it on the - // success path. `structural_superseded` is empty here (only - // persist_freemap fills it); clear defensively. - self.structural_reuse = self.pending_structural_frees.clone(); - self.structural_superseded.clear(); - // R1: clone the live-slot counts and reset the insert cursor. - // The cursor is always None at begin — it only tracks pages - // allocated during the current transaction. - self.current_live_slots = self.committed_live_slots.clone(); - self.insert_cursor = None; - self.active_txn = true; - self.savepoints.clear(); - self.txn_freed_pages.clear(); - Ok(()) - } - - /// Durably commit the active transaction. - /// - /// Commit protocol — ORDERING IS LOAD-BEARING. Each numbered step encodes a - /// specific crash-safety guarantee; reordering any of them can lose data or - /// expose torn state on recovery. - /// - /// A commit issues THREE fsyncs, not two: a pre-drain (step 0) plus the two - /// numbered below. Both pre-drain and step 1 are part of the "all data - /// durable BEFORE the superblock" phase — the pre-drain just moves some of - /// that flushing earlier; the superblock fsync (step 4) is the second phase. - /// - /// 0. Pre-drain the page cache (I28). BEFORE step 1, `commit_inner` flushes - /// the cache once so that `persist_freemap`'s own page allocation cannot - /// trip the spill / `CacheFull` ceiling mid-commit (which would poison on - /// an operational error). This is the FIRST of the three fsyncs. See the - /// I28 comment in `commit_inner` for why it is conditional-safe. - /// - /// 1. Flush all dirty data pages to disk AND fsync. - /// PageCache::flush() writes every dirty page then calls fsync(). After - /// this returns, every page the new superblock will reference is durable - /// on the storage medium. WHY FIRST: the new superblock is the pointer - /// that makes these pages "live". If we wrote the superblock before the - /// data pages were durable and crashed, recovery would pick up a - /// superblock whose root_handle_table_page points into a page whose - /// contents were never persisted — corruption with a valid checksum on - /// the superblock but garbage at the referenced page. - /// - /// 2. Compute the new superblock in memory. - /// Bump txn_counter first so (a) the new superblock outranks the old one - /// via Superblock::select()'s max_by_key, and (b) `txn_counter % - /// superblock_count` selects which slot to overwrite (step 3). For N=2 - /// this is the original parity alternation; for N>=3 (R4) it is true - /// round-robin across all N slots. total_pages is queried from the file - /// AFTER flush() so any new_page() allocations are reflected. - /// - /// 3. Write the new superblock to the INACTIVE slot. - /// The target is `txn_counter % superblock_count`, which always - /// points at the stalest slot. The N-1 other slots (including the - /// previously-active one, at counter txn_counter-1) are untouched - /// and still hold valid superblocks at strictly smaller counters. - /// WHY: if we crash during this write, the target slot may be torn - /// (bad checksum) but every other slot still holds the last - /// committed state (or earlier ones). Recovery picks the highest - /// surviving valid counter and the transaction is simply lost — - /// never half-applied. Overwriting an active slot in place would - /// be catastrophic: a torn write there could destroy a valid - /// superblock. Higher N buys survival of CONSECUTIVE torn writes - /// to the same target slot on retry (see `create_new` docstring). - /// - /// 4. fsync the superblock write. - /// This is the LINEARIZATION POINT of the commit. Before this fsync the - /// transaction is not durable, even if write_page returned; the kernel - /// may still be holding the superblock page in its buffer cache. After - /// this fsync returns successfully, a crash-and-recover will observe the - /// new state. A SINGLE fsync (combining data pages and superblock) would - /// be unsafe because the OS is free to reorder writes within an fsync - /// boundary — the superblock could reach the disk before the data pages - /// it references, creating a window where a crash leaves a valid-looking - /// superblock pointing at non-durable data. - /// - /// 5. Update in-memory committed_roots and clear txn state. - /// Only after the superblock fsync succeeds do we promote current_roots - /// to committed_roots. If ANY step in the protocol fails the manager is - /// poisoned (see the I1 block below) — active_txn / committed_roots are - /// left untouched but no public API will accept further calls; the only - /// legal recovery is close + reopen, which picks the last-durable - /// superblock via `Superblock::select`. Retry-in-place is forbidden - /// because a half-committed state (dirty flags already cleared in the - /// cache, txn_counter possibly bumped, target slot possibly torn on - /// disk) cannot be safely continued, and Linux fsyncgate semantics make - /// re-calling fsync() after a failed fsync unsafe regardless. - pub fn commit(&mut self) -> Result<()> { - self.check_alive()?; - // Special poison policy for commit: we refuse BOTH operational and - // fatal errors that arise after the commit protocol has started. - // The operational NoActiveTransaction case is checked BEFORE any - // protocol state is touched, so it stays operational and does not - // poison. But once cache.flush() has run, any subsequent error — - // even an otherwise operational one — leaves the manager in a - // partial-commit state (dirty flags cleared in the cache, counter - // possibly bumped, superblock possibly torn on disk) that cannot be - // safely continued. Under Linux fsyncgate semantics a failed fsync - // cannot be retried at all, so we poison and force the caller to - // reopen. - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - let result = self.commit_inner(); - if result.is_err() { - self.poisoned.set(true); - } - result - } - - fn commit_inner(&mut self) -> Result<()> { - // I27: flatten every still-active savepoint's `freed_pages` - // back into `txn_freed_pages` before persist_freemap consumes - // it. savepoint_inner moves `txn_freed_pages` INTO the - // savepoint record (via std::mem::take), so any frees that - // happened before a still-unreleased savepoint otherwise get - // dropped on the floor when step 5 calls `savepoints.clear()` - // — a permanent freemap leak for the "commit with savepoint - // active" pattern. Mirrors `release_inner`'s merge but applied - // across the full stack. We take the lists out of the - // savepoints (rather than iterating by reference) so the - // savepoints hold no stale `freed_pages` if we ever change - // step 5 to drain instead of clear; current code is equivalent - // either way. - for sp in self.savepoints.iter_mut() { - self.txn_freed_pages.append(&mut sp.freed_pages); - } - - // I28: drain the page cache BEFORE persist_freemap runs. Without - // this, `persist_freemap`'s own `allocate_data_page` can trip - // `maybe_evict`'s spill-or-CacheFull decision (every existing entry - // dirty, nothing evictable, and either spillway disabled or full) - // and return `ChiselError::CacheFull` or `ChiselError::SpillwayFull`. - // The CacheFull variant is operational-by-design (I19 docs: "caller - // recovers by - // committing or rolling back"), but commit's poison wrapper fires - // on any error once the protocol has started — demoting an - // operational signal to fatal for a caller who has no legal - // action left (commit is precisely what failed). Pre-draining - // clears every dirty pin so the ceiling is reachable via normal - // eviction when persist_freemap itself allocates. Cost: one - // extra fsync on every commit. That is consistent with the - // project's explicit "durability over performance" posture — - // the alternative reclassifies CacheFull as fatal inside commit, - // which is both more surprising and harder to document cleanly. - // - // Ordering note: this flush is safe to do before persist_freemap. - // The shadow-paging invariant requires "new-freemap-page durable - // before superblock" (step 1's flush does that). The pre-drain - // only affects user-dirty pages, which are already part of the - // transaction's durable write set — just flushed earlier. The - // subsequent step 1 flush handles the one new freemap page - // persist_freemap adds. - self.cache.borrow_mut().flush()?; - - // Step 0 (ISSUES.md R2 / I11): persist the freemap tree. This marks - // `txn_freed_pages` (plus the prior commit's deferred structural frees) - // free in a COW of the committed tree and updates - // `current_roots.{freemap_page, freemap_depth}`. Runs BEFORE the main - // flush so the new freemap pages join the same durable write set as all - // other dirty data pages. - self.persist_freemap()?; - - // Hold one RefMut for the remaining steps. Dropping and - // re-borrowing between steps would be semantically identical - // but noisier. - let mut cache = self.cache.borrow_mut(); - - // Step 1: Flush all dirty pages (PageCache::flush internally fsyncs). - // After this, every page the new superblock will reference is on disk. - cache.flush()?; - - // Step 2: Build the new superblock. Bumping txn_counter here both makes - // it outrank the current superblock on recovery AND (via parity) picks - // the target slot in step 3. - // - // I119 (ISSUES.md, 2026-06-21): checked, not `+= 1`. A wrapped counter - // would corrupt `Superblock::select`'s "highest counter wins" (release - // wrap to 0) — far worse than the loud, controlled panic here. Overflow - // needs 2^64 commits, so it is structurally unreachable; a dedicated - // fatal error variant for an impossible event would be speculative - // public surface, so the `expect` on the invariant is proportionate. - self.txn_counter = self - .txn_counter - .checked_add(1) - .expect("txn_counter overflowed u64 (2^64 commits) — unreachable"); - let total_pages = cache.file_page_count()?; - let sb = Superblock { - magic: page::MAGIC, - format_version: page::FORMAT_VERSION, - txn_counter: self.txn_counter, - root_handle_table_page: self.current_roots.handle_table_page, - root_freemap_page: self.current_roots.freemap_page, - total_pages, - next_handle: self.current_roots.next_handle, - page_size: PAGE_SIZE as u32, - named_roots: self.current_roots.named_roots, - // R4: every slot records the current N so open-time - // recovery can discover it from the winning slot without - // external hints. - superblock_count: self.superblock_count, - root_membership_index_page: self.current_roots.membership_index_page, - // Freemap tree depth, paired with root_freemap_page. 0 = today's - // single-leaf format; grows as the tree deepens. - freemap_depth: self.current_roots.freemap_depth, - }; - let buf = sb.serialize(); - // Step 3: Write to the INACTIVE slot. For N superblock slots, - // the slot is `txn_counter % N` — a round-robin that always - // targets the stalest slot. With N=2 this is the parity - // alternation from the original layout; with N>=3 it extends - // to true round-robin. The currently-active slot (and every - // other non-target slot) is never touched, so a torn write - // here can only damage the new superblock, never the N-1 - // last-known-good ones. - let inactive = self.txn_counter % self.superblock_count as u64; - cache.io_mut().write_page(inactive, &buf)?; - // Step 4: Durability linearization point. Until this fsync returns the - // transaction is not crash-safe; after it returns the new state is - // observable on recovery. - cache.io_mut().fsync()?; - - // Step 5: Promote in-memory state. Only now is the txn officially committed. - self.committed_roots = self.current_roots.clone(); - self.committed_roots.total_pages = total_pages; - // The committed freemap tree advances automatically: its {root, depth} - // ride in current_roots, promoted into committed_roots just above. No - // separate in-memory freemap copy to advance. - // R1: promote the live-slot counts. The cursor is per-transaction - // and gets reset for the next begin(). - self.committed_live_slots = self.current_live_slots.clone(); - self.insert_cursor = None; - self.active_txn = false; - self.savepoints.clear(); - // txn_freed_pages were already marked free in the new committed freemap - // tree by persist_freemap; clear the vector now that it's done its job. - self.txn_freed_pages.clear(); - // Every freemap page COW'd this transaction is now committed; the next - // transaction must COW (not edit in place) any of them it touches. - self.freemap_session_owned.clear(); - // Promote the freemap structural recycle for the next transaction: the - // pages this commit superseded (`structural_superseded`) become dead the - // instant the superblock flips above — and the reuse-pool remainder - // (`structural_reuse` ids not consumed as COW targets) is likewise still - // dead and reusable. Both become next transaction's `pending_structural_frees`. - self.pending_structural_frees.clear(); - self.pending_structural_frees - .append(&mut self.structural_superseded); - self.pending_structural_frees - .append(&mut self.structural_reuse); - - Ok(()) - } - - /// Abort the active transaction and discard all in-memory changes. - /// - /// Uses watermark-based rollback (ISSUES.md I3): `cache.truncate` is - /// called with `committed_roots.total_pages`, which both drops every - /// cache entry for pages allocated during the transaction AND truncates - /// the file back to its pre-transaction size. This fixes the earlier - /// bug where rollback would leave zeroed trailing pages in the file - /// because the cache-level discard did not propagate to `ftruncate`. - /// - /// Because `PageCache::new_page()` hands out monotonically increasing - /// ids, the pre-transaction watermark cleanly separates "pages that - /// existed at begin() time" (< watermark, preserved) from "pages - /// allocated during this transaction" (>= watermark, discarded). No - /// per-page tracking list is required. - pub fn rollback(&mut self) -> Result<()> { - self.check_alive()?; - let result = self.rollback_inner(); - self.poison_on_fatal(result) - } - - fn rollback_inner(&mut self) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - - // Rollback the cache in two steps: - // (a) Discard every dirty entry. This catches pages REUSED from - // the freemap whose id is less than the watermark — the - // watermark-based truncate below only catches extended - // pages. After discard, the next read for such a page id - // will re-load the last-committed content from disk, which - // is exactly the pre-transaction state. Safe because - // `flush()` (commit) always clears dirty flags, so any - // dirty entry was created in the current transaction. - // (b) Truncate to committed_roots.total_pages. This rewinds - // next_page_id AND shrinks the file, dropping every page - // allocated via extension (id >= watermark). Together with - // (a), this returns the cache and file to their exact - // pre-transaction state. - { - let mut cache = self.cache.borrow_mut(); - cache.discard_all_dirty(); - cache.truncate(self.committed_roots.total_pages)?; - } - - self.current_roots = self.committed_roots.clone(); - // C1: MembershipIndex.outer_depth is in-memory state that index grows - // mutate during the transaction, but it is NOT carried in Roots, so the - // snapshot restore above does not rewind it. Re-derive it from the (now - // committed) root — mirroring the open-time recovery — so the in-memory - // descent depth matches the page it descends. Otherwise handles_with_tag - // mis-descends a rolled-back-shallow root with a stale-deep depth. - { - let mut cache = self.cache.borrow_mut(); - let depth = - RadixU64::recover_depth(&mut cache, self.current_roots.membership_index_page)?; - self.membership_index.set_outer_depth(depth); - } - // I99: HandleTable.depth is the same kind of in-memory radix-depth cache - // as outer_depth above -- mutated by grows, not carried in Roots -- so it - // must also be re-derived from the restored root. Otherwise a rolled-back - // handle-table grow leaves the descent depth too deep and lookups - // mis-descend, returning InvalidHandle for committed handles. - { - let mut cache = self.cache.borrow_mut(); - let depth = - HandleTable::recover_depth(&mut cache, self.current_roots.handle_table_page)?; - self.handle_table.set_depth(depth); - } - // The freemap root+depth were restored by `current_roots = - // committed_roots.clone()` above; any dirty freemap pages this - // transaction COW'd sit above the watermark and were dropped by the - // truncate. The hint is untracked, so nothing to revert. - // - // `pending_structural_frees` is left intact: begin() CLONED it into - // `structural_reuse` rather than moving it, so it still holds the - // pre-transaction dead-freemap-page set — correct, since a rolled-back - // transaction's structural recycle is exactly the pre-transaction one. - // We DISCARD the in-transaction structural working state: - // * `structural_superseded` holds committed-tree freemap pages this - // aborted transaction COW'd-over; the abort means the committed tree - // still references them, so they are NOT dead and must never be - // recycled. - // * `structural_reuse` was the working copy; drop it. - // * the session-owned set: any freemap pages this aborted transaction - // COW'd sit above the watermark and were just truncated, so their ids - // must not be treated as in-place-mutable next transaction. - self.structural_superseded.clear(); - self.structural_reuse.clear(); - self.freemap_session_owned.clear(); - // R1: revert the live-slot counts and drop the insert cursor. - self.current_live_slots = self.committed_live_slots.clone(); - self.insert_cursor = None; - self.active_txn = false; - self.savepoints.clear(); - self.txn_freed_pages.clear(); - Ok(()) - } - - /// Push a named savepoint onto the stack. Captures the current - /// `next_page_id` watermark so `rollback_to(name)` can truncate the - /// cache back to this exact point. `freed_pages` is moved INTO the - /// savepoint record so the enclosing transaction's `txn_freed_pages` - /// accumulates only frees from the savepoint's own scope. - pub fn savepoint(&mut self, name: &str) -> Result<()> { - self.check_alive()?; - let result = self.savepoint_inner(name); - self.poison_on_fatal(result) - } - - fn savepoint_inner(&mut self, name: &str) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - if self.savepoints.iter().any(|sp| sp.name == name) { - return Err(ChiselError::DuplicateSavepoint(name.to_string())); - } - let watermark = self.cache_watermark(); - // R1: snapshot the live-slot map and the cursor. Also drop the - // cursor in the active scope — once a savepoint exists, the - // insert path stops packing into the cursor (same posture as - // freemap reuse: savepoints disable the optimization so the - // rollback_to semantics stay simple). - let live_slots = self.current_live_slots.clone(); - let insert_cursor = self.insert_cursor; - self.insert_cursor = None; - self.savepoints.push(Savepoint { - name: name.to_string(), - roots: self.current_roots.clone(), - watermark, - freed_pages: std::mem::take(&mut self.txn_freed_pages), - live_slots, - insert_cursor, - }); - Ok(()) - } - - /// Roll back to a named savepoint without ending the transaction. - /// Truncates the cache to the savepoint's watermark (discarding every - /// page allocated after the savepoint), restores the roots snapshot, - /// and pops any savepoints layered on top. The named savepoint itself - /// remains on the stack and can be rolled back to again or released. - /// - /// NOTE: `freed_pages` from savepoints layered on top (and from - /// `self.txn_freed_pages`) are dropped here, which is correct — - /// those frees never became durable, and the roots/page contents - /// those frees described have been rewound along with the cache - /// truncate. Post-R2, `commit()` DOES return freed pages to the - /// freemap; this rollback path simply discards the unfinished - /// accounting. - pub fn rollback_to(&mut self, name: &str) -> Result<()> { - self.check_alive()?; - let result = self.rollback_to_inner(name); - self.poison_on_fatal(result) - } - - fn rollback_to_inner(&mut self, name: &str) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - let idx = self - .savepoints - .iter() - .position(|sp| sp.name == name) - .ok_or_else(|| ChiselError::SavepointNotFound(name.to_string()))?; - - let watermark = self.savepoints[idx].watermark; - self.cache.borrow_mut().truncate(watermark)?; - - self.current_roots = self.savepoints[idx].roots.clone(); - // C1: re-derive outer_depth from the restored savepoint root (see rollback_inner). - { - let mut cache = self.cache.borrow_mut(); - let depth = - RadixU64::recover_depth(&mut cache, self.current_roots.membership_index_page)?; - self.membership_index.set_outer_depth(depth); - } - // I99: re-derive handle-table depth from the restored savepoint root - // (same rationale as rollback_inner / outer_depth). - { - let mut cache = self.cache.borrow_mut(); - let depth = - HandleTable::recover_depth(&mut cache, self.current_roots.handle_table_page)?; - self.handle_table.set_depth(depth); - } - // R1: restore live-slot counts and cursor from the savepoint - // snapshot. The cursor was force-cleared when the savepoint was - // created, so this sets the cursor back to whatever value it - // held BEFORE the savepoint was taken (typically also None, - // since savepoint-bearing transactions disable packing). - self.current_live_slots = self.savepoints[idx].live_slots.clone(); - self.insert_cursor = self.savepoints[idx].insert_cursor; - self.savepoints.truncate(idx + 1); - self.txn_freed_pages.clear(); - - Ok(()) - } - - /// Release (flatten) a named savepoint and everything layered on top - /// of it. Under watermark-based rollback, this is just `savepoints - /// .truncate(idx)` plus a merge of freed-page lists — the released - /// savepoints' allocated pages remain reachable via the outer - /// watermark (i.e. `committed_roots.total_pages`), which is still the - /// correct rollback destination for the enclosing transaction. - pub fn release(&mut self, name: &str) -> Result<()> { - self.check_alive()?; - let result = self.release_inner(name); - self.poison_on_fatal(result) - } - - fn release_inner(&mut self, name: &str) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - let idx = self - .savepoints - .iter() - .position(|sp| sp.name == name) - .ok_or_else(|| ChiselError::SavepointNotFound(name.to_string()))?; - - // Merge freed_pages from all released savepoints back into the - // current transaction's list. This preserves the invariant that - // txn_freed_pages holds every "frees that would go to the freemap - // on commit" across the entire enclosing transaction, so a later - // rollback correctly drops them. - let mut merged_freed = Vec::new(); - for sp in self.savepoints[idx..].iter() { - merged_freed.extend_from_slice(&sp.freed_pages); - } - merged_freed.append(&mut self.txn_freed_pages); - - self.savepoints.truncate(idx); - self.txn_freed_pages = merged_freed; - - Ok(()) - } - - /// Insert a value and return a stable handle. - /// - /// Handles are dense u64s drawn from current_roots.next_handle, starting at - /// 1 — handle 0 is reserved as the "no handle" sentinel and is never - /// returned. Large values - /// (> MAX_INLINE_VALUE) go to an overflow chain and the HandleEntry records - /// the first overflow page directly; small values get a slot in a freshly - /// allocated data page. Either way the handle_table.insert() COWs the spine - /// from leaf to root and returns the new root page_id, which becomes the - /// new current_roots.handle_table_page. This is the fundamental shadow- - /// paging step: the old root is still reachable via committed_roots and is - /// untouched on disk until commit swaps the superblock. - pub fn allocate(&mut self, value: &[u8]) -> Result { - self.check_alive()?; - let result = self.allocate_inner(value, 0); - self.poison_on_fatal(result) - } - - pub fn allocate_tagged(&mut self, value: &[u8], tag: u32) -> Result { - self.check_alive()?; - let result = self.allocate_inner(value, tag); - self.poison_on_fatal(result) - } - - /// Compute the membership-index root produced by inserting `(tag, handle)` - /// WITHOUT installing it into `current_roots`; superseded pages are appended - /// to `freed`. Split out so `allocate_inner` can stage the forward- and - /// reverse-map updates and install them atomically (BUG#2). On error, - /// `MembershipIndex::insert` leaves `self.outer_depth` unchanged (it writes - /// back only on success), so the caller need not restore any reverse-map - /// in-memory depth. - fn membership_insert_candidate( - &mut self, - tag: u32, - handle: u64, - freed: &mut Vec, - ) -> Result { - let reuse = self.savepoints.is_empty(); - let mut tree = self.take_freemap_tree(); - let result = { - let hint = &mut self.freemap_hint; - let pool = &mut self.structural_reuse; - let mut cache = self.cache.borrow_mut(); - let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); - self.membership_index.insert( - &mut cache, - self.current_roots.membership_index_page, - tag, - handle, - &mut alloc, - freed, - ) - }; - // Freemap growth is installed into roots regardless of success: the tree - // pages were extended (never freed), so a non-fatal failure that discards - // `freed`/the candidate root leaves these extra pages as harmless - // above-watermark scratch, exactly like the other COW pages on an aborted - // prepare. put_freemap_tree drains the freemap COW supersedes into - // structural_superseded and returns the session set so the next site in - // this transaction stays in-place. - self.put_freemap_tree(tree); - result - } - - /// Compute the handle-table root produced by inserting `entry` for `handle` - /// WITHOUT installing it into `current_roots`; superseded spine pages are - /// appended to `freed`. The forward-map counterpart to - /// `membership_insert_candidate` for `allocate_inner`'s atomic staging - /// (BUG#2). NOTE: `HandleTable::insert` may `grow`, which bumps the - /// in-memory descent depth eagerly — the caller captures and restores that - /// depth on the prepare-abort path. - fn handle_table_insert_candidate( - &mut self, - handle: u64, - entry: &HandleEntry, - freed: &mut Vec, - ) -> Result { - // Test-only injection (see `fail_next_handle_table_op`): simulate a - // non-fatal CacheFull at the forward / handle-table step — the one - // carrying the eager depth bump for allocate. Lives here so BOTH - // allocate_inner and update_inner exercise the real abort/unwind. No - // production artifact under `#[cfg(not(test))]`. - #[cfg(test)] - if self.fail_next_handle_table_op.replace(false) { - return Err(ChiselError::CacheFull { limit: 0 }); - } - let reuse = self.savepoints.is_empty(); - let mut tree = self.take_freemap_tree(); - let result = { - let hint = &mut self.freemap_hint; - let pool = &mut self.structural_reuse; - let mut cache = self.cache.borrow_mut(); - let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); - self.handle_table.insert( - &mut cache, - self.current_roots.handle_table_page, - handle, - entry, - &mut alloc, - freed, - ) - }; - // Install freemap growth into roots (and return the session set) so the - // NEXT candidate in this allocate (the reverse-map insert) threads the - // up-to-date tree and treats already-COW'd freemap pages as in-place. The - // freemap COW supersedes go to structural_superseded via put_freemap_tree. - // See membership_insert_candidate for the abort-safety reasoning. - self.put_freemap_tree(tree); - result - } - - /// Unwind the installed state from a partially-completed `allocate_inner` - /// PREPARE phase after a non-fatal failure. - /// - /// WHAT IS RESTORED (the installed state is a no-op): - /// - `current_roots.handle_table_page` — reverts to the pre-allocate value, - /// undoing any lazy `ensure_handle_table` materialization (empty DB goes - /// back to `PAGE_ID_NONE`). - /// - `handle_table` descent depth — restored from the saved value, undoing - /// the eager bump that `HandleTable::grow` applies before its fallible COW. - /// - Inline value's data slot — released via `release_data_slot` so - /// `current_live_slots` and the insert cursor stay consistent with the - /// un-installed root. A page allocated solely for this value goes to zero - /// occupancy and is queued for reclamation; a shared cursor page keeps a - /// defrag-reclaimable dead slot (exactly like a normal delete). - /// - `next_handle` — never consumed (bumped only in the infallible INSTALL - /// phase), so there is nothing to undo here. - /// - /// WHAT IS NOT RESTORED (a bounded allocated-but-unreferenced residue): - /// - Freemap COW pages drawn during the candidate allocations. When reuse is - /// enabled, `cow_alloc` may have cleared free bits in the committed freemap - /// and advanced the tree to cover the candidate-spine pages, leaving those - /// ids allocated-but-unreferenced. Restoring them here would require either - /// re-marking them free (fighting the COW dirty-page I20 invariant) or - /// re-sorting and re-queuing them as freed data pages (introducing I20 - /// dirty-page hazards and growth regressions on the abnormal path). - /// Instead, the residue is reclaimed by the expected rollback - /// (`discard_all_dirty` + watermark truncate restore the freemap to its - /// committed state). Overflow value pages are in the same residue class. - /// The residue leaks only if the caller commits after the operational - /// error rather than rolling back — contrary to documented contract. - fn abort_allocate_prepare( - &mut self, - saved_root: u64, - saved_depth: u32, - inline_page: Option, - ) { - self.current_roots.handle_table_page = saved_root; - self.handle_table.set_depth(saved_depth); - if let Some(page_id) = inline_page { - self.release_data_slot(page_id); - } - } - - /// Compute the membership-index root produced by removing `(tag, handle)` - /// WITHOUT installing it; returns `(new_root, was_present)`. Counterpart to - /// `membership_insert_candidate` for `delete_inner`'s atomic staging (BUG#2). - fn membership_remove_candidate( - &mut self, - tag: u32, - handle: u64, - freed: &mut Vec, - ) -> Result<(u64, bool)> { - let reuse = self.savepoints.is_empty(); - let mut tree = self.take_freemap_tree(); - let result = { - let hint = &mut self.freemap_hint; - let pool = &mut self.structural_reuse; - let mut cache = self.cache.borrow_mut(); - let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); - self.membership_index.remove( - &mut cache, - self.current_roots.membership_index_page, - tag, - handle, - &mut alloc, - freed, - ) - }; - self.put_freemap_tree(tree); - result - } - - /// Test-only fault decision for the reverse-map (membership-index) step, - /// shared by `allocate_inner` and `delete_inner`. Returns true (inject a - /// non-fatal CacheFull) if the one-shot `fail_next_membership_op` is armed, - /// or if the `fail_membership_op_after` countdown reaches this op. Consuming - /// here keeps the injection logic in one place. - #[cfg(test)] - fn inject_membership_failure(&self) -> bool { - if self.fail_next_membership_op.replace(false) { - return true; - } - let remaining = self.fail_membership_op_after.get(); - if remaining == 0 { - return false; - } - self.fail_membership_op_after.set(remaining - 1); - remaining == 1 - } - - fn allocate_inner(&mut self, value: &[u8], tag: u32) -> Result { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - - // BUG#2 atomic staging: the FORWARD map (the chunk's HandleEntry.tag in - // the handle table) and the REVERSE map (tag -> handles in the - // membership index, powering handles_with_tag / delete-by-tag) must - // become durable together. We compute both candidate roots in a fallible - // PREPARE phase that never installs into `current_roots`, then install - // them together in an infallible phase. - // - // A non-fatal CacheFull/SpillwayFull mid-prepare is unwound by - // `abort_allocate_prepare`, which is a no-op for the INSTALLED state: - // neither forward nor reverse map changes, the eagerly-bumped - // handle-table depth and any lazily-created root are restored, the - // inline value's data slot is released (keeping live-slot / cursor - // accounting consistent), and the handle id is not consumed - // (next_handle is bumped only on success). - // - // What the abort does NOT restore is the freemap COW that the candidate - // allocations performed: when reuse is enabled, `cow_alloc` may have - // drawn candidate-spine pages from the freemap (clearing their bits and - // advancing the tree), and those page ids are now allocated-but- - // unreferenced. This is a BOUNDED residue — the same class as any - // post-allocation failure — fully reclaimed by the expected rollback - // (discard_all_dirty + watermark truncate restore the freemap to its - // committed state). It materializes as a leak only if the caller commits - // after the operational error instead of rolling back, which is contrary - // to the documented contract. - let handle = self.current_roots.next_handle; - - // Value storage (PREPARE). For an inline value this also bumps - // current_live_slots and may set the insert cursor; capture its page id - // so a later prepare failure can release it via `abort_allocate_prepare`, - // keeping that bookkeeping consistent with the un-installed root. - // Overflow storage has no live-slot/cursor side effects. - let mut inline_page: Option = None; - let entry = if value.len() > MAX_INLINE_VALUE { - let first_page = { - let mut cache = self.cache.borrow_mut(); - Overflow::write(&mut cache, value)? - }; - HandleEntry { - page_id: first_page, - slot_index: 0, - flags: HandleFlags::Overflow, - tag, - client_byte: 0, - } - } else { - let (data_page_id, slot) = self.insert_into_data_page(value)?; - inline_page = Some(data_page_id); - HandleEntry { - page_id: data_page_id, - slot_index: slot, - flags: HandleFlags::Live, - tag, - client_byte: 0, - } - }; - - // Capture the handle-table root/depth BEFORE `ensure_handle_table` may - // lazily materialize an empty root, so a prepare failure restores - // current_roots to its true pre-allocate state (an empty DB reverts to - // PAGE_ID_NONE). The depth capture also covers `HandleTable::grow`, which - // bumps the in-memory descent depth EAGERLY (before its fallible leaf - // COW) while we defer the root install. The membership index needs no - // such save: MembershipIndex::insert writes its outer_depth back only on - // the success path, so a failed reverse-map op never advances it. - let saved_ht_root = self.current_roots.handle_table_page; - let saved_ht_depth = self.handle_table.depth(); - self.ensure_handle_table()?; - - // FORWARD map: compute the new handle-table root; do NOT install yet. - // (handle_table_insert_candidate carries the #[cfg(test)] forward-step - // fault injection, shared with update_inner's handle-table step.) - let mut ht_freed: Vec = Vec::new(); - let ht_new_root = match self.handle_table_insert_candidate(handle, &entry, &mut ht_freed) { - Ok(r) => r, - Err(e) => { - self.abort_allocate_prepare(saved_ht_root, saved_ht_depth, inline_page); - return Err(e); - } - }; - - // REVERSE map: compute the new membership-index root; do NOT install - // yet. tag 0 = untagged (never indexed). - let mut mi_new_root: Option = None; - let mut mi_freed: Vec = Vec::new(); - if tag != 0 { - // Test-only injection (see `fail_next_membership_op`): simulate a - // non-fatal CacheFull at the reverse-map step so the regression test - // exercises the REAL failure handling below. No production artifact: - // the non-test `let res` is the only one compiled outside tests. - #[cfg(test)] - let res: Result = if self.inject_membership_failure() { - Err(ChiselError::CacheFull { limit: 0 }) - } else { - self.membership_insert_candidate(tag, handle, &mut mi_freed) - }; - #[cfg(not(test))] - let res: Result = self.membership_insert_candidate(tag, handle, &mut mi_freed); - - match res { - Ok(r) => mi_new_root = Some(r), - Err(e) => { - // The handle-table insert above already succeeded and may - // have grown the tree (bumping the in-memory depth) and - // produced a candidate root we are now discarding. Unwind the - // whole prepare so current_roots, the depth, and the inline - // value's slot all return to their pre-allocate state. - self.abort_allocate_prepare(saved_ht_root, saved_ht_depth, inline_page); - return Err(e); - } - } - } - - // INSTALL phase (infallible): both maps move together, and only now is - // the handle id consumed and the superseded spine pages queued for - // reclamation at commit. - self.current_roots.next_handle += 1; - self.current_roots.handle_table_page = ht_new_root; - self.txn_freed_pages.append(&mut ht_freed); - if let Some(root) = mi_new_root { - self.current_roots.membership_index_page = root; - self.txn_freed_pages.append(&mut mi_freed); - } - - Ok(handle) - } - - pub fn tag(&self, handle: u64) -> Result { - self.check_alive()?; - let result = self.tag_inner(handle); - self.poison_on_fatal(result) - } - - /// The handle-table root for the *current read view*: the in-progress - /// `current_roots` while a transaction is active (read-your-own-writes), - /// otherwise the last durably-committed `committed_roots`. Returns - /// `PAGE_ID_NONE` for an empty database — read paths guard on that - /// before walking the tree. Centralizes the snapshot selection shared - /// by every read-path helper (`tag`, `client_byte`, `read`, `handles`, - /// `handle_live_page_id`). - fn live_handle_table_root(&self) -> u64 { - if self.active_txn { - self.current_roots.handle_table_page - } else { - self.committed_roots.handle_table_page - } - } - - /// Look up a handle that must be live, applying the "deleted ⇒ - /// `InvalidHandle`" rule in ONE place (I125). Every read/mutation entry - /// point that needs a live `HandleEntry` — `read`, `tag`, `client_byte`, - /// `set_client_byte`, `update`, `delete_tagged` — goes through here, so the - /// liveness invariant cannot drift between callers. - /// - /// `handle_table::lookup` already collapses a tombstone (and an empty/absent - /// tree, via `live_handle_table_root` returning `PAGE_ID_NONE`) to `None`, - /// so the `ok_or` below is the single site that raises the operational - /// `InvalidHandle`. Callers that want "absent is not an error" (e.g. - /// `handle_live_page_id`, which returns `Ok(None)`) deliberately do NOT use - /// this and keep their own Option-returning lookup. - fn lookup_live(&self, handle: u64) -> Result { - let root = self.live_handle_table_root(); - let mut cache = self.cache.borrow_mut(); - self.handle_table - .lookup(&mut cache, root, handle)? - .ok_or(ChiselError::InvalidHandle(handle)) - } - - fn tag_inner(&self, handle: u64) -> Result { - Ok(self.lookup_live(handle)?.tag) - } - - /// Return the opaque client byte stored in `handle`'s entry. Returns 0 if - /// never set (including every chunk created before this feature). Rejects - /// deleted handles with `InvalidHandle` via the shared `lookup_live` guard - /// (I125 — `read`, `tag`, and `delete_tagged` apply the identical rule). - /// Takes `&self`. - pub fn client_byte(&self, handle: u64) -> Result { - self.check_alive()?; - let result = self.client_byte_inner(handle); - self.poison_on_fatal(result) - } - - fn client_byte_inner(&self, handle: u64) -> Result { - Ok(self.lookup_live(handle)?.client_byte) - } - - /// Set the opaque client byte for `handle`. Requires an active - /// transaction; durable on commit, reverted on rollback. Any `u8` is - /// valid. COWs only the handle-table leaf — no data-page, overflow, or - /// membership-index work. Takes `&mut self`. - pub fn set_client_byte(&mut self, handle: u64, byte: u8) -> Result<()> { - self.check_alive()?; - let result = self.set_client_byte_inner(handle, byte); - self.poison_on_fatal(result) - } - - fn set_client_byte_inner(&mut self, handle: u64, byte: u8) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - let mut entry = self.lookup_live(handle)?; - entry.client_byte = byte; - self.ht_insert(handle, &entry)?; - Ok(()) - } - - pub fn handles_with_tag(&self, tag: u32) -> Result> { - self.check_alive()?; - let result = self.handles_with_tag_inner(tag); - self.poison_on_fatal(result) - } - - fn handles_with_tag_inner(&self, tag: u32) -> Result> { - let root = if self.active_txn { - self.current_roots.membership_index_page - } else { - self.committed_roots.membership_index_page - }; - let mut cache = self.cache.borrow_mut(); - // No PAGE_ID_NONE guard (unlike tag_inner): an empty/absent index is a - // legitimate "no handles with this tag" -> handles_for_tag returns an - // empty Vec for a PAGE_ID_NONE root, whereas a missing handle table is an error. - self.membership_index.handles_for_tag(&mut cache, root, tag) - } - - /// Read a value by handle. - /// - /// If a transaction is active, reads see the in-progress (uncommitted) state - /// through current_roots — i.e. "read your own writes". Otherwise reads go - /// through committed_roots, the last durably-committed snapshot. There is no - /// MVCC / snapshot isolation for concurrent readers because the writer is - /// single-threaded; this branch is purely about making the active writer - /// see its own pending mutations. - /// - /// F3: takes `&self`. Internally, the page cache is wrapped in a - /// RefCell so that the mutation required by LRU bookkeeping / page - /// loading can happen behind a shared reference. See the field-level - /// comment on `cache` for the full rationale and why RefCell was - /// chosen over Mutex. - pub fn read(&self, handle: u64) -> Result> { - self.check_alive()?; - let result = self.read_inner(handle); - self.poison_on_fatal(result) - } - - fn read_inner(&self, handle: u64) -> Result> { - let entry = self.lookup_live(handle)?; - - let mut cache = self.cache.borrow_mut(); - match entry.flags { - HandleFlags::Live => { - let buf = cache.get(entry.page_id)?; - // The handle-table entry insists this slot is live. If - // `DataPage::read` returns None anyway, the data page's - // structural state disagrees with the handle table — the - // page header, slot directory, or slot entry is damaged. - // That's CorruptPage (fatal / poisons the manager), not - // InvalidHandle (operational). - match DataPage::read(buf, entry.slot_index) { - Some(data) => Ok(data.to_vec()), - None => Err(ChiselError::CorruptPage { - page_id: entry.page_id, - }), - } - } - HandleFlags::Overflow => Overflow::read(&mut cache, entry.page_id), - // Unreachable in practice: `lookup_live` already excludes tombstones - // (I125). Kept as an exhaustive, non-panicking backstop — if the - // liveness invariant were ever violated, read still returns the - // operational `InvalidHandle` rather than aborting the writer. - HandleFlags::Deleted => Err(ChiselError::InvalidHandle(handle)), - } - } - - /// Update an existing handle to point at a new value. - /// - /// Allocates a new slot/overflow chain for the new value and rewrites - /// the HandleEntry via COW. The OLD location is retired differently - /// depending on its kind: - /// - /// * Inline (Live): goes through `release_data_slot`, which - /// decrements the per-page live-slot count (R1). Only when the - /// count reaches zero does the entire page land in - /// `txn_freed_pages`. Otherwise the slot becomes a tombstone, - /// reclaimable only via defrag (R3). - /// * Overflow: the whole chain is deleted and every page in the - /// chain is pushed onto `txn_freed_pages`. - /// - /// The earlier "assumes one live slot per page / must change when R1 - /// lands" caveat is OBSOLETE — R1 has landed and the slot-level - /// accounting below implements exactly the post-R1 contract. - pub fn update(&mut self, handle: u64, value: &[u8]) -> Result<()> { - self.check_alive()?; - let result = self.update_inner(handle, value); - self.poison_on_fatal(result) - } - - fn update_inner(&mut self, handle: u64, value: &[u8]) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - - // `update` requires an active txn (checked above), so `lookup_live`'s - // read-view root is `current_roots` — read-your-own-writes. - let entry = self.lookup_live(handle)?; - - // Atomic staging (same discipline as delete_inner): do NOT retire the - // OLD value's storage until the NEW entry is durably installed. The - // previous "free old first" ordering meant a non-fatal CacheFull during - // the new-value write or the handle-table install left the committed - // handle still pointing at pages already queued for reclamation — a - // reachable-but-free page that commit then frees, corrupting the live - // value on the next freemap reuse. We compute the new value, the new - // handle-table root, and the old-location free set in a fallible PREPARE - // phase that touches no installed state, then install the new entry and - // retire the old location together in an infallible INSTALL phase. A - // mid-prepare failure is a complete no-op. - // - // `update` replaces an EXISTING handle, so handle_table.insert never - // grows (handle < capacity) — no in-memory depth save is needed (unlike - // allocate_inner). - - // Test-only injection (see `fail_next_update_value_write`): now the FIRST - // fallible step, so a simulated failure here retires nothing. - #[cfg(test)] - if self.fail_next_update_value_write.replace(false) { - return Err(ChiselError::CacheFull { limit: 0 }); - } - - // PREPARE: write the new value storage. Tags and the client byte are - // entry-resident and carried forward unchanged (the handle is unchanged, - // so the membership index needs no edit — only the value's storage - // moves). Capture the inline data page so a later prepare failure can - // release it, keeping live-slot / cursor bookkeeping consistent. - let mut new_inline_page: Option = None; - let new_entry = if value.len() > MAX_INLINE_VALUE { - let first_page = { - let mut cache = self.cache.borrow_mut(); - Overflow::write(&mut cache, value)? - }; - HandleEntry { - page_id: first_page, - slot_index: 0, - flags: HandleFlags::Overflow, - tag: entry.tag, - client_byte: entry.client_byte, - } - } else { - let (data_page_id, slot) = self.insert_into_data_page(value)?; - new_inline_page = Some(data_page_id); - HandleEntry { - page_id: data_page_id, - slot_index: slot, - flags: HandleFlags::Live, - tag: entry.tag, - client_byte: entry.client_byte, - } - }; - - // PREPARE: compute the new handle-table root (no install). On failure, - // release the just-reserved inline slot so live-slot accounting stays - // consistent with the un-installed root (overflow new-value pages are the - // bounded commit-after-error leak class); the OLD location is untouched. - let mut ht_freed: Vec = Vec::new(); - let ht_new_root = - match self.handle_table_insert_candidate(handle, &new_entry, &mut ht_freed) { - Ok(r) => r, - Err(e) => { - if let Some(page_id) = new_inline_page { - self.release_data_slot(page_id); - } - return Err(e); - } - }; - - // PREPARE: compute the OLD location's free set without applying it. For - // Overflow this is a read-only walk of the old chain (a fallible - // cold-page load); for Live the page id is released in the install phase; - // Deleted carries no storage. On the overflow-walk failure, unwind the - // new inline slot — the old chain is untouched (the walk frees nothing). - enum OldRelease { - Inline(u64), - Overflow(Vec), - Nothing, - } - let old_release = match entry.flags { - HandleFlags::Live => OldRelease::Inline(entry.page_id), - HandleFlags::Overflow => { - let walked = { - let mut cache = self.cache.borrow_mut(); - Overflow::collect_chain_pages(&mut cache, entry.page_id) - }; - match walked { - Ok(freed) => OldRelease::Overflow(freed), - Err(e) => { - if let Some(page_id) = new_inline_page { - self.release_data_slot(page_id); - } - return Err(e); - } - } - } - HandleFlags::Deleted => OldRelease::Nothing, - }; - - // INSTALL phase (infallible): install the NEW entry first so the handle - // points at the new storage, THEN retire the OLD location (now genuinely - // unreferenced). For Live, release_data_slot does R1 slot accounting - // (freeing the page only when its last live slot goes); for Overflow, the - // walked chain pages are queued for reclamation. - self.current_roots.handle_table_page = ht_new_root; - self.txn_freed_pages.append(&mut ht_freed); - match old_release { - OldRelease::Inline(page_id) => self.release_data_slot(page_id), - OldRelease::Overflow(freed) => self.txn_freed_pages.extend_from_slice(&freed), - OldRelease::Nothing => {} - } - - Ok(()) - } - - /// Delete a handle. - /// - /// Retires the old location (inline slot via `release_data_slot` - /// with its R1 slot-level accounting; overflow chain by deleting - /// every page in the chain into `txn_freed_pages`) and then asks - /// the handle table to remove the mapping via COW. The earlier - /// "whole-page free assumes one value per page" caveat referenced - /// by `update`'s docstring is obsolete post-R1. - pub fn delete(&mut self, handle: u64) -> Result<()> { - self.check_alive()?; - let result = self.delete_inner(handle); - self.poison_on_fatal(result) - } - - fn delete_inner(&mut self, handle: u64) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - - // BUG#2 atomic staging (see allocate_inner): the FORWARD map (the - // handle-table tombstone) and the REVERSE map (membership-index removal) - // must become durable together. We compute both candidate roots — plus - // the fallible part of value-storage release — in a PREPARE phase that - // never touches `current_roots`, then install everything in an - // infallible phase. A non-fatal CacheFull/SpillwayFull mid-prepare - // leaves the delete a complete no-op, so the reverse index can never - // retain a member for a tombstoned handle — the stale entry that would - // otherwise later escalate to a fatal CorruptPage. - // - // No in-memory depth save is needed here (unlike allocate): handle-table - // DELETE never grows the tree, and MembershipIndex::remove writes its - // outer_depth back only on the success path. - - // FORWARD map: compute the tombstoned root; do NOT install yet. A single - // tree walk (I32) COWs the leaf with a tombstone and returns the - // previous entry. Returns (root, None) — unchanged root, no COW — if the - // handle was absent or already a tombstone; we escalate None to - // InvalidHandle to preserve the public-API behavior. - let mut ht_freed: Vec = Vec::new(); - let reuse = self.savepoints.is_empty(); - let mut tree = self.take_freemap_tree(); - let delete_result = { - let hint = &mut self.freemap_hint; - let pool = &mut self.structural_reuse; - let mut cache = self.cache.borrow_mut(); - let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); - self.handle_table.delete( - &mut cache, - self.current_roots.handle_table_page, - handle, - &mut alloc, - &mut ht_freed, - ) - }; - // Install freemap growth (supersedes go to structural_superseded). Done - // BEFORE the `?` so a delete that COW'd the freemap leaf yet then errored - // still records the extended root and returns the session set. - self.put_freemap_tree(tree); - let (ht_new_root, prev_entry) = delete_result?; - let entry = prev_entry.ok_or(ChiselError::InvalidHandle(handle))?; - - // Stage the value-storage release. The only FALLIBLE part — walking an - // overflow chain to collect its page ids — runs here in prepare; - // discarding the result on a later failure is safe (the still-current - // old entry keeps referencing the chain). The actual free-queueing / - // slot release is deferred to the install phase below. Order vs. the - // tombstone write does not matter for correctness — both become durable - // (or roll back) atomically at commit. - enum PendingRelease { - Inline(u64), - Overflow(Vec), - } - let release = match entry.flags { - HandleFlags::Live => PendingRelease::Inline(entry.page_id), - HandleFlags::Overflow => { - let freed = { - let mut cache = self.cache.borrow_mut(); - Overflow::collect_chain_pages(&mut cache, entry.page_id)? - }; - PendingRelease::Overflow(freed) - } - HandleFlags::Deleted => { - // I45 (ISSUES.md, 2026-05-22): a Deleted entry that `ok_or` - // didn't catch means the in-memory state contradicts itself — - // handle_table::delete returns None for already-tombstoned - // handles and the ok_or above converts None into the typed - // error, so reaching this arm signals a broken cross-module - // contract (most likely a future refactor of delete). Surface - // it typed rather than aborting the caller's process. Returned - // BEFORE any install, so current_roots stays untouched. - return Err(ChiselError::CorruptPage { - page_id: entry.page_id, - }); - } - }; - - // REVERSE map: compute the membership-removed root; do NOT install yet. - // The tag comes from the tombstoned entry. Tag 0 (untagged) is never in - // the index, so there is nothing to remove. - let mut mi_new_root: Option = None; - let mut idx_freed: Vec = Vec::new(); - if entry.tag != 0 { - // Test-only injection (see `inject_membership_failure`): simulate a - // non-fatal CacheFull at the reverse-map step so regression tests - // exercise the REAL failure handling below. No production artifact: - // the non-test `let res` is the only one compiled outside tests. - #[cfg(test)] - let res: Result<(u64, bool)> = if self.inject_membership_failure() { - Err(ChiselError::CacheFull { limit: 0 }) - } else { - self.membership_remove_candidate(entry.tag, handle, &mut idx_freed) - }; - #[cfg(not(test))] - let res: Result<(u64, bool)> = - self.membership_remove_candidate(entry.tag, handle, &mut idx_freed); - - let (new_index_root, removed) = res?; - // A tagged live handle always carries a reverse-index entry: - // allocate_tagged installs it atomically (BUG#2 / PR #40) and tags - // are immutable. So its removal must report present; a false here - // means the forward and reverse maps diverged from some OTHER - // source. Debug-only on purpose: a file committed while BUG#2 was - // still live (pre-#40) could carry a real on-disk divergence, and - // the open path gates MAJOR version only — such a legacy file must - // stay openable and a delete of its diverged handle must remain - // recoverable, so this never gates release builds. - debug_assert!( - removed, - "membership index diverged: tagged handle {handle} (tag {}) had no reverse entry", - entry.tag - ); - mi_new_root = Some(new_index_root); - } - - // INSTALL phase (infallible): tombstone + value release + reverse-map - // removal all become visible together. The superseded handle-table - // spine pages are queued only now, post-install, so an early prepare - // failure leaves them referenced by the still-current old tree. - self.current_roots.handle_table_page = ht_new_root; - self.txn_freed_pages.append(&mut ht_freed); - match release { - PendingRelease::Inline(page_id) => self.release_data_slot(page_id), - PendingRelease::Overflow(freed) => self.txn_freed_pages.extend_from_slice(&freed), - } - if let Some(root) = mi_new_root { - self.current_roots.membership_index_page = root; - self.txn_freed_pages.append(&mut idx_freed); - } - - Ok(()) - } - - pub fn delete_tagged(&mut self, handle: u64, tag: u32) -> Result<()> { - self.check_alive()?; - let result = self.delete_tagged_inner(handle, tag); - self.poison_on_fatal(result) - } - - fn delete_tagged_inner(&mut self, handle: u64, tag: u32) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - // Lookup-then-delete: verify the tag before mutating anything, so a wrong - // tag leaves both the chunk and the membership index untouched. The extra - // lookup walk (delete_inner walks again) is the price of verify-before-mutate. - // `lookup_live` rejects an absent/tombstoned handle with `InvalidHandle` - // (I125) BEFORE the tag comparison — a dead handle never surfaces as - // TagMismatch. - let actual = self.lookup_live(handle)?.tag; - if actual != tag { - return Err(ChiselError::TagMismatch { - handle, - expected: tag, - actual, - }); - } - self.delete_inner(handle) - } - - /// Bounded relation drop. See `Chisel::delete_with_tag` for the full - /// contract. Error semantics: a mid-pass `delete_inner` failure propagates - /// `Err` and the partial `TagDropProgress` is dropped — the deleted-this- - /// pass set is not reported. Each `delete_inner` is atomic (BUG#2 staging), - /// so the surviving in-transaction state is consistent (rollback or commit - /// are both safe); only the progress *reporting* is lost on error. - pub fn delete_with_tag(&mut self, tag: u32, max: usize) -> Result<(Vec, bool)> { - self.check_alive()?; - let result = self.delete_with_tag_inner(tag, max); - self.poison_on_fatal(result) - } - - fn delete_with_tag_inner(&mut self, tag: u32, max: usize) -> Result<(Vec, bool)> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - if max == 0 { - return Ok((Vec::new(), false)); - } - // Bounded enumeration: ask for max+1 so the count tells us whether more - // remain (len > max => not complete). saturating_add keeps the absurd - // max == usize::MAX case meaning "enumerate everything" instead of - // wrapping to 0 — which would enumerate nothing and falsely report the - // tag complete. The members snapshot is taken BEFORE the deletions, then - // each is deleted via delete_inner (which removes it from the index and - // frees its chunk). - // - // The snapshot must be MATERIALIZED into an owned Vec, not a live - // iterator: each delete_inner COWs the membership-index root (via - // membership_remove_candidate), so walking the index while deleting from - // it would descend a tree being rewritten underneath the walk. Collecting - // first decouples enumeration from the mutation it drives. - let members = { - let root = self.current_roots.membership_index_page; - let mut cache = self.cache.borrow_mut(); - self.membership_index.handles_for_tag_bounded( - &mut cache, - root, - tag, - max.saturating_add(1), - )? - }; - let complete = members.len() <= max; - let take: Vec = members.into_iter().take(max).collect(); - for &h in &take { - self.delete_inner(h)?; - } - Ok((take, complete)) - } - - /// Delete many handles in a single transaction. - /// - /// Today: this is a loop over `delete_inner`. After PR-A's fusion - /// (I32), each delete walks the handle table once per handle. For - /// dense delete patterns (many handles in the same leaf), a - /// per-leaf batched implementation would walk once per leaf - /// instead — that's tracked as I33 in ISSUES.md, deferred until - /// a workload demonstrates the win is worth the complexity. - /// - /// Error semantics: on the first error the loop stops and returns - /// the error. Handles deleted before the failure remain marked - /// for deletion in `current_roots`, so the caller can choose - /// between `rollback()` (abandon the whole batch) or `commit()` - /// (keep the partial work). - pub fn delete_many(&mut self, handles: &[u64]) -> Result<()> { - self.check_alive()?; - let result = self.delete_many_inner(handles); - self.poison_on_fatal(result) - } - - fn delete_many_inner(&mut self, handles: &[u64]) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - // See I33 in ISSUES.md for the deferred per-leaf batching work. - for &handle in handles { - self.delete_inner(handle)?; - } - Ok(()) - } - - /// Iterate over all live handles. - /// - /// F3: takes `&self` (same rationale as `read`). - pub fn handles(&self) -> Result> { - self.check_alive()?; - let result = self.handles_inner(); - self.poison_on_fatal(result) - } - - fn handles_inner(&self) -> Result> { - let root = self.live_handle_table_root(); - if root == PAGE_ID_NONE { - return Ok(Vec::new()); - } - let mut cache = self.cache.borrow_mut(); - let entries = self.handle_table.iter_live(&mut cache, root)?; - Ok(entries.into_iter().map(|(h, _)| h).collect()) - } - - /// Snapshot the four engine-activity counters (cache hits/misses, - /// pages allocated, fsync calls). Counters are cumulative from the - /// most recent open; the bench harness reads-subtract-reads for - /// per-cell deltas. Takes `&self` (F3); poison-aware via - /// `check_alive`. - pub fn counters(&self) -> Result { - self.check_alive()?; - Ok(self.cache.borrow().counters()) - } - - /// I74 (ISSUES.md, 2026-05-22): peek the spillway's current - /// (logical_bytes, max_bytes) for `Chisel::stats`. `None` if the - /// spillway has never been opened. Routes through the same - /// poison check as `counters()` so a fatal-error state surfaces - /// here too — operators reading `stats()` get a `Poisoned` - /// error rather than stale-looking Some(0,0). - pub fn spillway_capacity(&self) -> Result> { - self.check_alive()?; - Ok(self.cache.borrow().spillway_capacity()) - } - - // --- Named roots (ISSUES.md F2) --- - // - // The named-root table lives inside the superblock (see - // `superblock::NamedRoot`). Modifications update - // `current_roots.named_roots` in memory; on commit that array is - // copied into the new Superblock and fsync'd along with the rest. - // On rollback or `rollback_to`, the usual snapshot restore reverts - // named roots alongside the handle-table root — no extra plumbing. - // - // Name validation is intentionally strict: names must be non-empty, - // must fit in NAMED_ROOT_NAME_LEN bytes, must not contain NUL - // (because NUL is the "empty slot" sentinel), and must be valid - // UTF-8 at the API boundary. Names are compared byte-for-byte after - // validation; the fixed 24-byte buffer is NUL-padded. - - /// Validate a root name and return its byte form, padded to - /// NAMED_ROOT_NAME_LEN with trailing NULs. Returns `InvalidRootName` - /// on any violation. - fn encode_root_name(name: &str) -> Result<[u8; NAMED_ROOT_NAME_LEN]> { - let bytes = name.as_bytes(); - if bytes.is_empty() || bytes.len() > NAMED_ROOT_NAME_LEN { - return Err(ChiselError::InvalidRootName); - } - if bytes.contains(&0) { - return Err(ChiselError::InvalidRootName); - } - let mut encoded = [0u8; NAMED_ROOT_NAME_LEN]; - encoded[..bytes.len()].copy_from_slice(bytes); - Ok(encoded) - } - - /// Bind `name` to `handle` in the named-root table. If `name` already - /// exists, its handle is overwritten. If it doesn't exist and the - /// table has no empty slots, returns `RootNameTableFull`. Requires an - /// active transaction and becomes durable on commit; reverts on - /// rollback/rollback_to. - pub fn set_root_name(&mut self, name: &str, handle: u64) -> Result<()> { - self.check_alive()?; - let result = self.set_root_name_inner(name, handle); - self.poison_on_fatal(result) - } - - fn set_root_name_inner(&mut self, name: &str, handle: u64) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - let encoded = Self::encode_root_name(name)?; - - // First pass: update in place if the name already exists. - for entry in self.current_roots.named_roots.iter_mut() { - if !entry.is_empty() && entry.name == encoded { - entry.handle = handle; - return Ok(()); - } - } - // Second pass: install in the first empty slot. - for entry in self.current_roots.named_roots.iter_mut() { - if entry.is_empty() { - entry.name = encoded; - entry.handle = handle; - return Ok(()); - } - } - Err(ChiselError::RootNameTableFull) - } - - /// Look up a named root. Returns `Ok(None)` if the name is not bound. - /// Reads see the transactional view: inside an active transaction, - /// pending `set_root_name` / `clear_root_name` changes are visible; - /// outside a transaction, reads the last durably committed table. - /// - /// Takes `&self` — named-root reads are semantically read-only. - pub fn get_root_name(&self, name: &str) -> Result> { - self.check_alive()?; - let result = self.get_root_name_inner(name); - self.poison_on_fatal(result) - } - - fn get_root_name_inner(&self, name: &str) -> Result> { - let encoded = Self::encode_root_name(name)?; - let table = if self.active_txn { - &self.current_roots.named_roots - } else { - &self.committed_roots.named_roots - }; - for entry in table.iter() { - if !entry.is_empty() && entry.name == encoded { - return Ok(Some(entry.handle)); - } - } - Ok(None) - } - - /// Remove a named root. No-op if the name is not bound (returns Ok). - /// Requires an active transaction. Becomes durable on commit; - /// reverts on rollback/rollback_to. - pub fn clear_root_name(&mut self, name: &str) -> Result<()> { - self.check_alive()?; - let result = self.clear_root_name_inner(name); - self.poison_on_fatal(result) - } - - fn clear_root_name_inner(&mut self, name: &str) -> Result<()> { - if !self.active_txn { - return Err(ChiselError::NoActiveTransaction); - } - let encoded = Self::encode_root_name(name)?; - for entry in self.current_roots.named_roots.iter_mut() { - if !entry.is_empty() && entry.name == encoded { - *entry = NamedRoot::EMPTY; - return Ok(()); - } - } - Ok(()) - } - - /// Poisoning-aware wrapper around `PageCache::file_page_count`. Called - /// by `Chisel::stats()` so that a fatal I/O error while measuring the - /// file size also poisons the manager. - /// - /// F3: takes `&self`. - pub fn file_page_count(&self) -> Result { - self.check_alive()?; - let result = self.cache.borrow_mut().file_page_count(); - self.poison_on_fatal(result) - } - - // --- Selective defragmentation support (ISSUES.md R3 + I17) --- - // - // These methods expose just enough of the R1 live-slot tracking - // for `defrag::defrag` to do selective page compaction. The - // defrag module is in-crate and could in principle access the - // fields directly, but going through named methods keeps the - // intent obvious at each call site. - - /// Page ids of data pages whose effective density (live slots / - /// stored slots) is strictly less than `threshold_ratio`. A - /// freshly-packed page with every slot still live has density - /// 1.0; a page that originally packed 39 values but now has only - /// 5 live (34 dead-weight tombstones) has density 0.128 and is a - /// strong defrag candidate. - /// - /// The metric uses the page's OWN stored-slot count (read from - /// the on-disk header via `DataPage::slot_count`) as the - /// denominator — not the max-observed count in the database — - /// because dead-weight slots are what defrag is trying to reclaim. - /// The older "relative to densest" metric failed for the case of a - /// single remaining sparse page (density 1.0 against itself). - /// - /// Returns an empty set when `threshold_ratio <= 0`. Fallible - /// because the per-page stored count is read through the cache. - pub fn sparse_data_pages( - &self, - threshold_ratio: f64, - ) -> Result> { - self.check_alive()?; - let result = self.sparse_data_pages_inner(threshold_ratio); - self.poison_on_fatal(result) - } - - fn sparse_data_pages_inner( - &self, - threshold_ratio: f64, - ) -> Result> { - let mut sparse = std::collections::HashSet::new(); - if threshold_ratio <= 0.0 { - return Ok(sparse); - } - let page_ids: Vec = self.current_live_slots.keys().copied().collect(); - for page_id in page_ids { - let live = match self.current_live_slots.get(&page_id) { - Some(&n) if n > 0 => n, - _ => continue, - }; - let stored = { - let mut cache = self.cache.borrow_mut(); - DataPage::slot_count(cache.get(page_id)?) as u32 - }; - if stored == 0 { - continue; - } - let density = live as f64 / stored as f64; - if density < threshold_ratio { - sparse.insert(page_id); - } - } - Ok(sparse) - } - - /// Snapshot of the page ids currently tracked as holding at least - /// one live slot. Used by `defrag::defrag` for the I17 stat: after - /// the sweep, `pages_freed` is the count of ids that were in this - /// snapshot and are no longer in `current_live_slots` — i.e., - /// pages that the sweep fully drained and returned to the freemap. - /// Net change in the live data-page count is the wrong metric here - /// because a relocation simultaneously drains a sparse page and - /// creates a dense one; the former should count as "reclaimed" - /// even when the latter offsets the net count. - pub fn data_page_ids_snapshot(&self) -> std::collections::HashSet { - self.current_live_slots.keys().copied().collect() - } - - /// Look up the data page id that currently holds `handle`. Returns - /// `Ok(None)` if the handle doesn't exist, is deleted, or points - /// at an overflow chain (for which the notion of "data page" does - /// not apply). - /// - /// Takes `&self`; uses the RefCell around the cache to perform the - /// handle-table lookup. Poisons the manager on fatal I/O or - /// checksum errors. - pub fn handle_live_page_id(&self, handle: u64) -> Result> { - self.check_alive()?; - let result = self.handle_live_page_id_inner(handle); - self.poison_on_fatal(result) - } - - fn handle_live_page_id_inner(&self, handle: u64) -> Result> { - let root = self.live_handle_table_root(); - if root == PAGE_ID_NONE { - return Ok(None); - } - let mut cache = self.cache.borrow_mut(); - let entry = match self.handle_table.lookup(&mut cache, root, handle)? { - Some(e) => e, - None => return Ok(None), - }; - if entry.flags == HandleFlags::Live { - Ok(Some(entry.page_id)) - } else { - Ok(None) - } - } - - /// The handle-table root page id of the active transaction's - /// in-progress roots. Used by `defrag::defrag` to short-circuit - /// the empty-database fast path. - /// - /// I39 (ISSUES.md, 2026-05-22): replaces a `pub fn current_roots() - /// -> (u64, u64, u64)` tuple return that exposed three fields when - /// only one was ever read. YAGNI: if a future caller wants - /// `freemap_page` or `next_handle`, add a sibling accessor at that - /// time rather than guessing the API shape now. `pub(crate)` - /// because `defrag` is the sole intended caller (transaction - /// module became `pub(crate)` in I35). - pub(crate) fn current_handle_table_root_page(&self) -> u64 { - self.current_roots.handle_table_page - } - - pub fn is_active(&self) -> bool { - self.active_txn - } - - pub fn set_cache_max_bytes(&mut self, bytes: u64) -> Result<()> { - self.check_alive()?; - if self.active_txn { - return Err(ChiselError::TransactionInProgress); - } - self.cache.borrow_mut().set_cache_max_bytes(bytes) - } - - pub fn set_spillway_max_bytes(&mut self, bytes: u64) -> Result<()> { - self.check_alive()?; - if self.active_txn { - return Err(ChiselError::TransactionInProgress); - } - self.cache.borrow_mut().set_spillway_max_bytes(bytes) - } - - pub fn set_drain_insertion(&mut self, policy: crate::DrainInsertion) -> Result<()> { - self.check_alive()?; - if self.active_txn { - return Err(ChiselError::TransactionInProgress); - } - // I40: PageCache::set_drain_insertion is now infallible (returns - // `()`). The poison + active-txn checks above are the real - // failure modes; we promote PageCache's `()` to `Ok(())` here. - self.cache.borrow_mut().set_drain_insertion(policy); - Ok(()) - } - - // --- Private helpers --- - - /// Release one slot from a data page (ISSUES.md R1). Decrements - /// `current_live_slots[page_id]`; if the count reaches zero, the - /// whole page becomes unreferenced and is pushed to - /// `txn_freed_pages` so commit can return it to the freemap. - /// Otherwise the slot becomes a tombstone: dead weight inside a - /// still-live page, reclaimable only via defrag. - /// - /// If the page is somehow not tracked in `current_live_slots` (a - /// bug; open-time scan should catch every live data page), this is - /// a no-op — we prefer leaking to a spurious free. - /// - /// NOTE: a stray orphaned line "Lazily create a handle table root - /// on first insert. A fresh database has" previously sat at the - /// top of this doc block (an interleaved remnant of - /// `ensure_handle_table`'s docstring); removed 2026-04-17 during - /// the commenting pass. The counterpart ("root_handle_table_page - /// == PAGE_ID_NONE; we don't materialize...") still sits above - /// `ensure_handle_table` below — both belong together. - fn release_data_slot(&mut self, page_id: u64) { - let Some(count) = self.current_live_slots.get_mut(&page_id) else { - return; - }; - if *count > 0 { - *count -= 1; - } - if *count == 0 { - self.current_live_slots.remove(&page_id); - // If this page is the active insert cursor, clear the - // cursor — it's about to become free space, and we don't - // want future inserts to pack into it and then find it - // disappearing at commit time. - if self.insert_cursor == Some(page_id) { - self.insert_cursor = None; - } - self.txn_freed_pages.push(page_id); - } - } - - /// Lazily create a handle table root on first insert. A fresh - /// database has `root_handle_table_page == PAGE_ID_NONE`; we don't - /// materialize the root until there is a handle to put in it, so - /// empty databases never pay for a handle-table page. No per-page - /// rollback bookkeeping — the watermark rollback mechanism (I3) - /// handles any page allocated here automatically. - fn ensure_handle_table(&mut self) -> Result<()> { - if self.current_roots.handle_table_page == PAGE_ID_NONE { - let root = { - let mut cache = self.cache.borrow_mut(); - self.handle_table.create_root(&mut cache)? - }; - self.current_roots.handle_table_page = root; - } - Ok(()) - } - - /// Place a value in a data page and return (page_id, slot_index). - /// - /// Post-R1 packing model: the transaction maintains an "insert - /// cursor" — a data page allocated earlier in THIS transaction - /// that still has space — and packs successive small-value inserts - /// into it until it fills. When the cursor is absent/full, a new - /// page is allocated (via `allocate_data_page`, which prefers - /// freemap reuse over file extension — R2) and becomes the new - /// cursor. Packing is disabled while savepoints are active: the - /// cursor is force-cleared by `savepoint()` and is NOT set when a - /// new page is allocated inside a savepoint scope, so each insert - /// under a savepoint gets its own page (the pre-R1 behavior). This - /// keeps the per-savepoint snapshot cheap to restore. - /// - /// Checksum is stamped eagerly after every mutation so the page carries a - /// valid internal checksum before any path could write it to the main - /// file — either the `flush` `write_page` at commit, or a spill-then-drain - /// write (an LRU-pressured dirty page is spilled to the spillway and later - /// drained back out to the main file). The next cold-load - /// (`page_cache::load_page`) verifies that checksum. - /// - /// Note: the spillway *transfer* does NOT rely on this. `rehydrate` - /// verifies the spillway's own per-slot checksum (`spillway::slot_checksum`), - /// never the page's internal bytes 8184..8192 — so a spilled page round-trips - /// safely whether or not its internal checksum is current. The internal - /// checksum only matters on the way to the main file. - /// - /// I78 proposes deferring this re-stamp to flush/drain time so a packed page - /// is hashed once, not once per value (a large bulk-insert win on fast - /// storage). It is deferred pending a benchmark; the difficulty is exactly - /// the spill-then-drain path, which would then have to re-stamp before the - /// main-file write. See ISSUES.md. - /// - /// Live-slot bookkeeping: every successful insert increments - /// `current_live_slots[page_id]`. `delete`/`update` consult this - /// map (via `release_data_slot`) to decide when a page is fully - /// empty and can be freed back to the freemap on commit. The map - /// is kept purely in memory — storing a slot count ON the data - /// page would force a COW (and a handle-table rewrite for every - /// entry pointing into it) on every delete. - fn insert_into_data_page(&mut self, value: &[u8]) -> Result<(u64, u16)> { - // Packing path: try to reuse the current cursor page if it - // has room. The cursor only exists when savepoints are empty - // (see savepoint_inner) so this branch implicitly respects - // the "no packing under savepoints" rule. - if let Some(cursor_page_id) = self.insert_cursor { - let slot_option = { - let mut cache = self.cache.borrow_mut(); - let buf = cache.get_mut(cursor_page_id)?; - let result = DataPage::insert(buf, value); - if result.is_some() { - page::stamp_checksum(buf); - } - result - }; - if let Some(slot) = slot_option { - *self.current_live_slots.entry(cursor_page_id).or_insert(0) += 1; - return Ok((cursor_page_id, slot)); - } - // Cursor page is full. Fall through to allocate a new one; - // the new page becomes the new cursor. - } - - // Allocate a fresh data page. Under active savepoints, the - // cursor stays None (set below, then cleared by the savepoint - // check in subsequent calls) so each insert gets its own page — - // matching the pre-R1 "one value per page" behavior within - // savepoint scopes, which is the price of keeping rollback_to - // semantics simple. - let page_id = self.allocate_data_page()?; - let slot = { - let mut cache = self.cache.borrow_mut(); - let buf = cache.get_mut(page_id)?; - DataPage::init_page(buf); - // I46 INVARIANT: DataPage::insert can only return None for - // "no room"; the page was just init'd via DataPage::init_page - // (empty), and the value's length was already checked against - // MAX_INLINE_VALUE upstream (the overflow path catches anything - // larger before we get here). If DataPage::insert ever grows - // other failure modes, this expect needs to translate them to - // typed errors instead of panicking. - let slot = DataPage::insert(buf, value).expect("value fits in empty page"); - page::stamp_checksum(buf); - slot - }; - - // Only install the new page as the cursor if we're outside any - // savepoint scope. During a savepoint scope the cursor stays - // None so packing is effectively disabled. - if self.savepoints.is_empty() { - self.insert_cursor = Some(page_id); - } - *self.current_live_slots.entry(page_id).or_insert(0) += 1; - Ok((page_id, slot)) - } -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::page_io::{Fault, PageIo}; - use tempfile::{NamedTempFile, TempDir}; - - fn fresh_manager() -> TransactionManager { - let file = NamedTempFile::new().unwrap(); - let io = PageIo::open(file.path(), false).unwrap(); - // Match Options::default()'s cache_max_bytes of 8 MiB (1024 pages) - // so tests that intentionally allocate many pages in a single - // transaction (e.g. the I3+I7 handle-table-growth test allocates - // 510+) stay well under the strict cache cap. spillway_max_bytes=0 - // preserves the legacy CacheFull-at-cap behavior in tests. - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let mut tm = TransactionManager::create_new(cache, 2).unwrap(); - // Commit once so there's a real baseline to read/write against. - tm.begin().unwrap(); - tm.commit().unwrap(); - tm - } - - /// C1 invariant: after a commit, NO page reachable from `committed_roots` - /// may be marked free in `committed_freemap`. A correct COW frees only - /// superseded pages; freeing a still-referenced page (the textbook C1 - /// violation — e.g. `grow` freeing the reparented old root, or `update` - /// freeing the OLD value before the new entry is installed) shows up here as - /// a page that is both reachable and free. This is deterministic regardless - /// of the freemap's lowest-id-first selection order, which makes black-box - /// reopen tests unreliable for catching C1. - /// - /// Reachability covers BOTH the index spines (handle-table + membership - /// outer/inner) AND the value storage every live handle points at (its - /// inline data page or its full overflow chain). The value-storage half is - /// essential: a spine-only walk cannot catch a value-page premature-free. - /// - /// PRECONDITION: call only between transactions (right after a commit), where - /// `committed_roots == current_roots` and the in-memory `handle_table.depth` - /// matches the committed root. `iter_live` descends with that live depth, so - /// calling this mid-transaction after a grow would mis-descend. - fn assert_no_reachable_page_is_free(tm: &TransactionManager) { - let mut reachable = Vec::new(); - { - let mut cache = tm.cache.borrow_mut(); - tm.handle_table - .collect_page_ids( - &mut cache, - tm.committed_roots.handle_table_page, - &mut reachable, - ) - .unwrap(); - tm.membership_index - .collect_page_ids( - &mut cache, - tm.committed_roots.membership_index_page, - &mut reachable, - ) - .unwrap(); - - // Value storage reachable through each live HandleEntry. - if tm.committed_roots.handle_table_page != PAGE_ID_NONE { - let live = tm - .handle_table - .iter_live(&mut cache, tm.committed_roots.handle_table_page) - .unwrap(); - for (_handle, entry) in live { - match entry.flags { - HandleFlags::Live => reachable.push(entry.page_id), - HandleFlags::Overflow => { - let chain = - Overflow::collect_chain_pages(&mut cache, entry.page_id).unwrap(); - reachable.extend(chain); - } - HandleFlags::Deleted => {} - } - } - } - } - // Query freeness through the committed freemap TREE (reconstructed from - // {root, depth}) rather than a flat in-memory bitmap — same C1 invariant, - // new storage representation. - let mut cache = tm.cache.borrow_mut(); - let tree = FreeMapTree::from_roots( - tm.committed_roots.freemap_page, - tm.committed_roots.freemap_depth, - ); - for id in reachable { - assert!( - !tree.is_free(&mut cache, id).unwrap(), - "page {id} is reachable from committed_roots but marked FREE in \ - the committed freemap — a still-referenced page was freed (C1 violation)" - ); - } - } - - // COW page reclamation must never free a page still referenced by the - // committed tree, even after the trees GROW (the reparenting paths). Forces - // a handle-table grow (>510 handles) and a membership inner-tree grow - // (>1021 members under one tag), then churns with reclamation, asserting the - // C1 invariant after every commit. - #[test] - fn reclamation_never_frees_a_reachable_page_after_grow() { - let mut tm = fresh_manager(); - let tag = 9u32; - let mut handles = Vec::new(); - let mut v: u32 = 0; - - // Build >1021 tagged members in small batches (stay under the 1024-page - // cache cap), forcing both trees to grow to depth >= 1. - for _ in 0..12 { - tm.begin().unwrap(); - for _ in 0..100 { - let h = tm.allocate_tagged(&v.to_le_bytes(), tag).unwrap(); - handles.push(h); - v += 1; - } - tm.commit().unwrap(); - assert_no_reachable_page_is_free(&tm); - } - assert!( - handles.len() > 1021, - "workload must exceed one membership leaf to force an inner grow" - ); - - // Churn with reclamation across committed transactions: update relocates - // the value and COWs the handle-table spine (freeing the old spine); - // set_client_byte COWs only the leaf. Batched per ~100 handles so a - // single transaction's dirty COW pages stay under the cache cap (within - // a txn, this-txn frees are not yet reusable). Re-check after each commit. - for round in 0..12u32 { - for (chunk_idx, chunk) in handles.chunks(100).enumerate() { - tm.begin().unwrap(); - for (j, h) in chunk.iter().enumerate() { - if (round as usize + chunk_idx + j) % 2 == 0 { - tm.set_client_byte(*h, round as u8).unwrap(); - } else { - tm.update(*h, &round.to_le_bytes()).unwrap(); - } - } - tm.commit().unwrap(); - assert_no_reachable_page_is_free(&tm); - } - } - - // Every handle still carries its tag and is enumerable after the churn. - for h in &handles { - assert_eq!(tm.tag(*h).unwrap(), tag); - } - assert_eq!(tm.handles_with_tag(tag).unwrap().len(), handles.len()); - } - - #[test] - fn tagged_membership_survives_rolled_back_outer_grow() { - let mut tm = fresh_manager(); - // Commit a small tag; the outer (tag-keyed) tree stays depth 0. - tm.begin().unwrap(); - let h = tm.allocate_tagged(b"keep", 3).unwrap(); - tm.commit().unwrap(); - assert_eq!(tm.handles_with_tag(3).unwrap(), vec![h]); - // New txn: a tag >= 1021 forces the outer tree to grow (depth 0 -> 1). - // Roll back. The grown root is discarded and current_roots snaps back to - // the depth-0 committed root; outer_depth must be restored to match. - tm.begin().unwrap(); - let _ = tm.allocate_tagged(b"discard", 5000).unwrap(); - tm.rollback().unwrap(); - // The committed small tag must still be readable (was silently lost before the fix). - assert_eq!( - tm.handles_with_tag(3).unwrap(), - vec![h], - "rolled-back outer grow corrupted committed membership" - ); - assert_eq!(tm.tag(h).unwrap(), 3); - // The discarded tag is gone. - assert_eq!(tm.handles_with_tag(5000).unwrap(), Vec::::new()); - } - - #[test] - fn tagged_membership_survives_rollback_to_savepoint() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let h = tm.allocate_tagged(b"keep", 7).unwrap(); - tm.savepoint("sp").unwrap(); - // Grow the outer tree past depth 0 inside the savepoint, then roll back to it. - let _ = tm.allocate_tagged(b"discard", 6000).unwrap(); - tm.rollback_to("sp").unwrap(); - // Still inside the active txn: the pre-savepoint tag must remain readable. - assert_eq!(tm.handles_with_tag(7).unwrap(), vec![h]); - assert_eq!(tm.handles_with_tag(6000).unwrap(), Vec::::new()); - tm.commit().unwrap(); - assert_eq!(tm.handles_with_tag(7).unwrap(), vec![h]); - } - - // Regression test for ISSUES.md I1. Once the manager is poisoned, - // every public entry point must return ChiselError::Poisoned rather - // than attempting the operation. This is the core invariant of the - // poison model — the test asserts it for each method independently - // so a future refactor that forgets to wrap a new entry point will - // fail loudly. - #[test] - fn poisoned_manager_rejects_every_public_entry_point() { - let mut tm = fresh_manager(); - tm.force_poison_for_test(); - assert!(tm.is_poisoned()); - - assert!(matches!(tm.begin(), Err(ChiselError::Poisoned))); - assert!(matches!(tm.commit(), Err(ChiselError::Poisoned))); - assert!(matches!(tm.rollback(), Err(ChiselError::Poisoned))); - assert!(matches!(tm.savepoint("x"), Err(ChiselError::Poisoned))); - assert!(matches!(tm.rollback_to("x"), Err(ChiselError::Poisoned))); - assert!(matches!(tm.release("x"), Err(ChiselError::Poisoned))); - assert!(matches!(tm.allocate(b"v"), Err(ChiselError::Poisoned))); - assert!(matches!(tm.read(0), Err(ChiselError::Poisoned))); - assert!(matches!(tm.update(0, b"v"), Err(ChiselError::Poisoned))); - assert!(matches!(tm.delete(0), Err(ChiselError::Poisoned))); - assert!(matches!(tm.handles(), Err(ChiselError::Poisoned))); - assert!(matches!(tm.file_page_count(), Err(ChiselError::Poisoned))); - assert!(matches!( - tm.allocate_tagged(b"v", 1), - Err(ChiselError::Poisoned) - )); - assert!(matches!(tm.tag(0), Err(ChiselError::Poisoned))); - assert!(matches!(tm.handles_with_tag(1), Err(ChiselError::Poisoned))); - assert!(matches!(tm.delete_tagged(0, 1), Err(ChiselError::Poisoned))); - assert!(matches!( - tm.delete_with_tag(1, 10), - Err(ChiselError::Poisoned) - )); - assert!(matches!(tm.client_byte(0), Err(ChiselError::Poisoned))); - assert!(matches!( - tm.set_client_byte(0, 1), - Err(ChiselError::Poisoned) - )); - } - - #[test] - fn fatal_error_outside_commit_also_poisons() { - // I112: a REAL fatal IoError on a cold read OUTSIDE any transaction - // poisons the manager (the non-commit fatal path, poison_on_fatal). This - // replaces the old force_poison_for_test() tautology with an injected - // fault. We reopen over the committed file so read(h) is a cache MISS - // that actually reaches read_page(pid). - let file = NamedTempFile::new().unwrap(); - let h; - let pid; - { - let io = PageIo::open(file.path(), false).unwrap(); - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let mut tm = TransactionManager::create_new(cache, 2).unwrap(); - tm.begin().unwrap(); - h = tm.allocate(b"durable").unwrap(); - tm.commit().unwrap(); - pid = tm.handle_live_page_id(h).unwrap().expect("live data page"); - } - - // Reopen: cold cache, so read(h) misses and calls read_page(pid). - let io = PageIo::open(file.path(), false).unwrap(); - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let tm = TransactionManager::open_existing(cache).unwrap(); - tm.cache.borrow().io().arm_fault(Fault::FailReadPage(pid)); - let result = tm.read(h); - assert!( - matches!(result, Err(ChiselError::IoError(_))), - "cold read fault must surface IoError, got {result:?}" - ); - assert!( - tm.is_poisoned(), - "a fatal read error outside commit must poison" - ); - } - - // Regression test for ISSUES.md I3 + I7. A transaction that forces - // handle-table growth allocates many pages (the data pages for each - // value, the handle-table leaves, the COW spine clones, and the - // new interior root from grow()). After rollback, every one of those - // pages must be gone — both from the in-memory cache AND from the - // file itself. - // - // Pre-I7, the old per-page dirty list missed intermediate COW pages. - // Pre-I3, rollback only discarded cache entries without truncating - // the file, so the extended pages leaked permanently. This test - // exercises both conditions in one shot by asserting the - // `next_page_id` watermark and the cache page-count return to their - // pre-transaction values after rollback. - #[test] - fn rollback_truncates_cache_and_file_to_pre_txn_watermark() { - let mut tm = fresh_manager(); - let pre_watermark = tm.cache.borrow().next_page_id(); - let pre_file_pages = tm.cache.borrow_mut().file_page_count().unwrap(); - - tm.begin().unwrap(); - tm.allocate(b"seed").unwrap(); - // Force handle-table growth by crossing the 510-entry leaf boundary. - for _ in 0..510 { - tm.allocate(b"f").unwrap(); - } - // Sanity: the transaction must have extended the cache past the - // pre-transaction watermark. Otherwise the test below is vacuous. - let mid_watermark = tm.cache.borrow().next_page_id(); - assert!( - mid_watermark > pre_watermark + 510, - "expected the transaction to allocate many pages beyond {pre_watermark}, got {mid_watermark}" - ); - - tm.rollback().unwrap(); - - let post_watermark = tm.cache.borrow().next_page_id(); - let post_file_pages = tm.cache.borrow_mut().file_page_count().unwrap(); - assert_eq!( - post_watermark, pre_watermark, - "rollback must rewind next_page_id to the pre-transaction watermark" - ); - assert_eq!( - post_file_pages, pre_file_pages, - "rollback must truncate the file back to its pre-transaction page count" - ); - } - - // rollback_to(name) must truncate cache+file to the savepoint's - // watermark, discarding every page allocated after the savepoint - // while preserving those allocated before it. This is the per- - // savepoint analogue of the full-rollback test above. - #[test] - fn rollback_to_savepoint_truncates_to_savepoint_watermark() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let h1 = tm.allocate(b"before").unwrap(); - tm.savepoint("sp").unwrap(); - let savepoint_watermark = tm.cache.borrow().next_page_id(); - let _h2 = tm.allocate(b"after").unwrap(); - let _h3 = tm.allocate(b"after-2").unwrap(); - assert!(tm.cache.borrow().next_page_id() > savepoint_watermark); - - tm.rollback_to("sp").unwrap(); - assert_eq!( - tm.cache.borrow().next_page_id(), - savepoint_watermark, - "rollback_to must rewind to the savepoint's watermark" - ); - // The pre-savepoint handle must still be readable. - assert_eq!(tm.read(h1).unwrap(), b"before"); - tm.commit().unwrap(); - } - - // An operational error (NoActiveTransaction, DuplicateSavepoint, - // InvalidHandle, etc.) must NOT poison. These are caller mistakes, - // not integrity failures — the manager stays usable. - #[test] - fn operational_error_does_not_poison() { - let mut tm = fresh_manager(); - - // NoActiveTransaction from commit — operational. - assert!(matches!(tm.commit(), Err(ChiselError::NoActiveTransaction))); - assert!(!tm.is_poisoned()); - - // NoActiveTransaction from allocate — operational. - assert!(matches!( - tm.allocate(b"v"), - Err(ChiselError::NoActiveTransaction) - )); - assert!(!tm.is_poisoned()); - - // DuplicateSavepoint — operational. - tm.begin().unwrap(); - tm.savepoint("a").unwrap(); - assert!(matches!( - tm.savepoint("a"), - Err(ChiselError::DuplicateSavepoint(_)) - )); - assert!(!tm.is_poisoned()); - - // InvalidHandle from read — operational. - assert!(matches!(tm.read(999), Err(ChiselError::InvalidHandle(_)))); - assert!(!tm.is_poisoned()); - } - - // Regression test for ISSUES.md I18. Inside commit_inner's - // persist_freemap step, the new-freemap-page allocation must never - // return an id that is still referenced by the currently-committed - // on-disk superblock. The two sources of such at-risk ids are: - // - // (1) `committed_roots.freemap_page` itself — the current - // on-disk freemap page; overwriting it mid-commit destroys - // the committed freemap snapshot. - // (2) Any id in `txn_freed_pages` — pages that held handle - // values reachable through the committed handle table; - // overwriting any of them mid-commit destroys a - // committed value. - // - // A crash in the window between `cache.flush()` and the superblock - // fsync would then leave the last-durable superblock pointing at - // a page whose bytes no longer match what it committed to — - // breaking the core shadow-paging invariant. The fix defers the - // merge of both at-risk sets into `current_freemap` until AFTER - // the new-freemap-page allocate has run, so `FreeMap::allocate_first` - // cannot return any of them during the vulnerable window. - #[test] - fn persist_freemap_does_not_reuse_committed_live_pages() { - let mut tm = fresh_manager(); - - // Commit 1: seed a non-trivial committed state. We need - // persist_freemap to actually materialize a freemap page, - // not take the early-exit path. That requires `txn_freed_pages` - // to be non-empty at commit, which means freeing at least one - // WHOLE data page — R1 slot packing keeps multi-slot data - // pages live even after individual deletes. The simplest way - // to guarantee whole-page frees is to use overflow-sized - // values (> MAX_INLINE_VALUE): each gets its own overflow - // chain, and delete releases every page in the chain into - // txn_freed_pages via Overflow::collect_chain_pages. - let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; - tm.begin().unwrap(); - let h_throwaway = tm.allocate(&big).unwrap(); - let h_live_a = tm.allocate(&big).unwrap(); - let h_live_b = tm.allocate(&big).unwrap(); - let h_live_c = tm.allocate(&big).unwrap(); - tm.delete(h_throwaway).unwrap(); - tm.commit().unwrap(); - - let committed_freemap_page = tm.committed_roots.freemap_page; - assert_ne!( - committed_freemap_page, PAGE_ID_NONE, - "test precondition: commit 1 should have established a freemap page" - ); - - // Commit 2: delete two more handles. release_data_slot pushes - // their data pages into `txn_freed_pages`; those pages are - // still referenced by commit 1's (currently-on-disk) - // superblock at the moment commit_inner runs persist_freemap. - tm.begin().unwrap(); - tm.delete(h_live_a).unwrap(); - tm.delete(h_live_b).unwrap(); - - let frozen_txn_freed: Vec = tm.txn_freed_pages.clone(); - assert!( - !frozen_txn_freed.is_empty(), - "test precondition: deletes should have populated txn_freed_pages" - ); - - tm.commit().unwrap(); - - // The at-risk set: anything that was still live under the - // prior committed superblock at the moment persist_freemap - // started allocating. - let mut still_live_pre_commit = frozen_txn_freed.clone(); - still_live_pre_commit.push(committed_freemap_page); - - let new_freemap_page = tm.committed_roots.freemap_page; - assert!( - !still_live_pre_commit.contains(&new_freemap_page), - "I18: persist_freemap allocated the new freemap page at an id \ - that was still referenced by the last-durable superblock. \ - new_freemap_page={new_freemap_page}, \ - committed_freemap_page was {committed_freemap_page}, \ - txn_freed_pages at commit time = {frozen_txn_freed:?}" - ); - - // Sanity: the un-deleted handle still reads back correctly - // (rules out a subtler corruption that survived the invariant - // check but poisoned the data plane). - assert_eq!(tm.read(h_live_c).unwrap(), big); - } - - // Regression test for ISSUES.md I27. `savepoint_inner` moves - // `txn_freed_pages` into the savepoint record via `std::mem::take`. - // If commit runs with savepoints still on the stack, the pre-fix - // `commit_inner` just called `self.savepoints.clear()` at step 5 - // and those `freed_pages` lists were dropped — never reaching the - // freemap. `persist_freemap` iterates only `self.txn_freed_pages`. - // The post-fix merge in commit_inner flattens every active - // savepoint's `freed_pages` back into `txn_freed_pages` before - // `persist_freemap` runs, so every page freed anywhere in the - // transaction reaches the committed freemap. - // - // Observable via `FreeMap::is_free(&committed_freemap, id)` — the - // freemap's public predicate avoids any reliance on subsequent - // allocator reuse (which depends on `savepoints.is_empty()` too and - // would muddy the test). - #[test] - fn commit_with_active_savepoint_returns_freed_pages_to_freemap() { - let mut tm = fresh_manager(); - - // Seed: enough overflow-sized handles that deleting them - // produces genuine page frees (R1 slot-packing would otherwise - // keep multi-slot data pages live). - let big: Vec = vec![0xCD; MAX_INLINE_VALUE + 32]; - tm.begin().unwrap(); - let h_a = tm.allocate(&big).unwrap(); - let h_b = tm.allocate(&big).unwrap(); - let h_keepalive = tm.allocate(&big).unwrap(); - tm.commit().unwrap(); - - // The leak pattern: delete first, THEN open a savepoint. The - // savepoint captures the accumulated `txn_freed_pages`, leaving - // the outer `txn_freed_pages` empty for the rest of the txn. - tm.begin().unwrap(); - tm.delete(h_a).unwrap(); - tm.delete(h_b).unwrap(); - let frozen_txn_freed: Vec = tm.txn_freed_pages.clone(); - assert!( - !frozen_txn_freed.is_empty(), - "test precondition: overflow-sized deletes should free at least one page" - ); - - tm.savepoint("s").unwrap(); - assert!( - tm.txn_freed_pages.is_empty(), - "savepoint_inner should have moved txn_freed_pages into the savepoint" - ); - - // Commit WITHOUT releasing the savepoint. Pre-fix this silently - // drops savepoint.freed_pages on `savepoints.clear()`; post-fix - // commit_inner merges them into txn_freed_pages first. - tm.commit().unwrap(); - - { - let mut cache = tm.cache.borrow_mut(); - let tree = FreeMapTree::from_roots( - tm.committed_roots.freemap_page, - tm.committed_roots.freemap_depth, - ); - for id in &frozen_txn_freed { - assert!( - tree.is_free(&mut cache, *id).unwrap(), - "I27: freed page {id} should be marked free in the committed \ - freemap tree after commit-with-active-savepoint; \ - frozen_txn_freed={frozen_txn_freed:?}" - ); - } - } - - // Sanity: the surviving handle still reads back (rules out a - // wider corruption that happens to also trip the is_free check). - assert_eq!(tm.read(h_keepalive).unwrap(), big); - } - - // ── Structural-page recycle: adversarial pin-tests ────────────────────── - // - // These three lock down the durability-critical freemap structural recycle - // (docs/specs/2026-06-22 "Structural-page reclamation"): the one-commit - // defer, the rollback reset of the recycle pools, and no lost/double free - // across reuse cycles. They are the GATE for the Phase 2 work — a violation - // is a crash-safety bug, not a cosmetic one. - - // Drive a commit that actually COWs the freemap and supersedes structural - // pages: allocate `n` overflow-sized values (each its own whole page chain), - // commit, then delete `del` of them and commit. The second commit's - // `persist_freemap` marks the freed pages free, COWing the freemap leaf/spine - // and superseding the old freemap pages — exactly the churn the recycle - // model is built around. Returns the surviving handles. - fn structural_churn(tm: &mut TransactionManager, big: &[u8], n: usize, del: usize) -> Vec { - tm.begin().unwrap(); - let mut handles: Vec = (0..n).map(|_| tm.allocate(big).unwrap()).collect(); - tm.commit().unwrap(); - - tm.begin().unwrap(); - for h in handles.drain(..del) { - tm.delete(h).unwrap(); - } - tm.commit().unwrap(); - handles - } - - // PROPERTY 1 — one-commit-defer crash-safety. - // - // A freemap page `P` superseded in transaction `T` is still referenced by - // the pre-`T` superblock until `T` commits, so it may be reused as a - // structural COW target ONLY starting in `T+1` (the one-commit defer). If - // `T+1` ever drew a COW target from a page it superseded THIS transaction, - // a crash before `T+1`'s superblock fsync would corrupt the page the - // recovered (pre-`T+1`) superblock still points at — a durability BUG. - // - // We capture the promoted recycle set at the START of the measured - // transaction `T+1` (== what `begin()` cloned into `structural_reuse`), then - // instrument every structural-reuse pop in `T+1` and assert each popped id is - // drawn from EXACTLY that promoted set — never an id minted or superseded - // within `T+1`. The thread-local reuse log records both pop sites - // (`structural_extend` and `persist_freemap`'s inline closure). - // - // Crucially, `T+1` is a WARMED-UP steady-state transaction doing many - // interleaved allocate+delete ops: each allocate reuses a bitmap-free data - // page, which COWs the freemap leaf in the transaction BODY and supersedes a - // freemap page mid-flight — so a later body allocation in the same - // transaction WOULD pop that just-superseded page if the defer were broken. - // (A single-op transaction supersedes the freemap only at persist_freemap, - // the last structural op, leaving no later pop to expose the bug — this test - // is structured to defeat that blind spot.) - #[test] - fn structural_recycle_one_commit_defer() { - let mut tm = fresh_manager(); - let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; - - // Warm up to steady state: a rotating live population so the bitmap holds - // free data pages (making body allocations COW the freemap) and the - // structural recycle is non-trivially populated. Run several - // delete-then-reallocate commits. - let mut live: Vec = Vec::new(); - tm.begin().unwrap(); - for _ in 0..16 { - live.push(tm.allocate(&big).unwrap()); - } - tm.commit().unwrap(); - for _ in 0..6 { - tm.begin().unwrap(); - let recycled: Vec = live.drain(..8).collect(); - for h in recycled { - tm.delete(h).unwrap(); - } - for _ in 0..8 { - live.push(tm.allocate(&big).unwrap()); - } - tm.commit().unwrap(); - } - - // Capture the promoted recycle the measured transaction inherits, and - // drain the warm-up's reuse log so only the measured transaction is seen. - let promoted: std::collections::HashSet = - tm.pending_structural_frees.iter().copied().collect(); - assert!( - !promoted.is_empty(), - "precondition: warm-up must leave a non-empty deferred recycle" - ); - let _ = take_structural_reuse_log(); - - // T+1 (measured): MANY interleaved allocate+delete ops. Each allocate - // claims a bitmap-free data page (COWing + superseding the freemap in the - // body), each delete frees a page; the heavy interleave means a freemap - // page superseded early in the body has many later body allocations that - // would pop it if the defer leaked same-txn supersedes into the pool. - tm.begin().unwrap(); - for _ in 0..6 { - let recycled: Vec = live.drain(..4).collect(); - for h in recycled { - tm.delete(h).unwrap(); - } - for _ in 0..4 { - live.push(tm.allocate(&big).unwrap()); - } - } - // Pages T+1 superseded so far (the body). persist_freemap adds more at - // commit; both must stay out of the reuse pops. - let superseded_in_t1: std::collections::HashSet = - tm.structural_superseded.iter().copied().collect(); - tm.commit().unwrap(); - - // Read the log IMMEDIATELY after T+1's commit — before any later - // transaction can pop from its OWN (legitimately) promoted recycle and - // pollute the capture with ids that were never inherited here. - let reused_in_t1 = take_structural_reuse_log(); - assert!( - !reused_in_t1.is_empty(), - "precondition: T+1 must actually reuse at least one deferred page \ - (else the defer is untested)" - ); - // NOTE: this block is a weak guard on its own. Under session-COW dedup a - // leaf is COW'd at most once per commit, so a same-transaction supersede - // and its only in-txn reuse pop are the SAME event — they cannot both be - // observed here. The load-bearing defer check is the cross-commit - // REACHABILITY assertion below; this block is kept as a cheap sanity rail. - for id in &reused_in_t1 { - assert!( - promoted.contains(id), - "one-commit-defer VIOLATION: T+1 reused freemap page {id} that was \ - NOT in the promoted recycle set {promoted:?} — it was minted or \ - superseded within T+1, so a pre-commit crash would corrupt the page \ - the last-durable superblock still references" - ); - assert!( - !superseded_in_t1.contains(id), - "one-commit-defer VIOLATION: T+1 reused page {id} that T+1 itself \ - superseded this transaction (still live under the last-durable \ - superblock) — reusing it pre-commit is a crash-safety bug" - ); - } - - // The defer's DURABLE consequence: a page T+1 superseded is now (post- - // commit) dead and queued for T+2 — but it must NOT be reachable in the - // just-committed live tree. A broken defer that re-routed a same-txn - // supersede into the reuse pool would surface here as a pool page still - // live in the committed tree (the corruption a pre-commit crash would - // expose). This is the cross-boundary half of the defer the in-txn pop - // check above cannot see (session-COW dedup COWs each leaf once, so the - // supersede and its only in-txn pop are the same event). - let reachable: std::collections::HashSet = { - let mut cache = tm.cache.borrow_mut(); - let committed = FreeMapTree::from_roots( - tm.committed_roots.freemap_page, - tm.committed_roots.freemap_depth, - ); - committed - .reachable_pages(&mut cache) - .unwrap() - .into_iter() - .collect() - }; - for id in &tm.pending_structural_frees { - assert!( - !reachable.contains(id), - "one-commit-defer VIOLATION: page {id} is queued for reuse in T+2 but is \ - still reachable in the committed freemap tree — a same-transaction \ - supersede leaked into the reuse pool while still live" - ); - } - } - - // PROPERTY 2 — rollback resets the recycle pools and session state. - // - // A rollback must leave the structural recycle exactly as a clean begin would - // see it: `structural_reuse` restored to the committed recycle state (derived - // from `pending_structural_frees`), `structural_superseded` cleared, and - // `freemap_session_owned` cleared. Leaking any of these into the next - // transaction would let it reuse a page the committed tree still references, - // or skip a needed COW on a now-committed page — both corruption. - #[test] - fn structural_recycle_rollback_resets_pools() { - let mut tm = fresh_manager(); - let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; - - // Establish a non-empty committed recycle so the test exercises a real - // restore target, not just emptiness. - let survivors = structural_churn(&mut tm, &big, 8, 4); - let committed_recycle: Vec = tm.pending_structural_frees.clone(); - assert!( - !committed_recycle.is_empty(), - "precondition: a prior commit must leave a non-empty deferred recycle" - ); - - // A transaction that mutates all three pools: allocations + deletes COW - // the freemap (filling session-owned + superseded), and the deletes' frees - // make persist-side reuse pops drain `structural_reuse`. Do NOT commit. - tm.begin().unwrap(); - let _fresh: Vec = (0..6).map(|_| tm.allocate(&big).unwrap()).collect(); - for h in &survivors { - tm.delete(*h).unwrap(); - } - // The session-owned set is populated by freemap COWs on the alloc/delete - // path; assert the test actually dirtied the state it is about to roll - // back (else the reset assertions are vacuous). - assert!( - !tm.freemap_session_owned.is_empty() || !tm.structural_superseded.is_empty(), - "precondition: the pre-rollback churn must have mutated freemap session/supersede state" - ); - - tm.rollback().unwrap(); - - // begin() CLONES `pending_structural_frees` into `structural_reuse`, so an - // aborted transaction's recycle is exactly the pre-transaction one: the - // committed recycle must be intact, and the working pools cleared. - assert_eq!( - tm.pending_structural_frees, committed_recycle, - "rollback must leave the committed deferred recycle intact" - ); - let recycle_after: std::collections::HashSet = - tm.pending_structural_frees.iter().copied().collect(); - let committed_set: std::collections::HashSet = - committed_recycle.iter().copied().collect(); - assert_eq!( - recycle_after, committed_set, - "the post-rollback recycle (what the next begin will seed structural_reuse from) \ - must equal the committed recycle state" - ); - assert!( - tm.structural_superseded.is_empty(), - "rollback must clear structural_superseded — those committed-tree pages are \ - still referenced and must never be recycled" - ); - assert!( - tm.freemap_session_owned.is_empty(), - "rollback must clear freemap_session_owned — leaking it would suppress a needed \ - COW and mutate a live committed page in place next transaction" - ); - - // Crucial follow-through: the next transaction must reuse ONLY the - // committed recycle, proving no aborted-transaction page leaked into the - // pool. (An aborted supersede leaking into reuse is a classic double-free.) - let _ = take_structural_reuse_log(); - tm.begin().unwrap(); - let mut next: Vec = (0..8).map(|_| tm.allocate(&big).unwrap()).collect(); - for h in survivors { - tm.delete(h).unwrap(); - } - tm.commit().unwrap(); - for id in take_structural_reuse_log() { - assert!( - committed_set.contains(&id), - "post-rollback transaction reused freemap page {id} not in the committed \ - recycle {committed_set:?} — rollback leaked structural pool state" - ); - } - for h in next.drain(..) { - tm.begin().unwrap(); - tm.delete(h).unwrap(); - tm.commit().unwrap(); - } - } - - // PROPERTY 2b — the orphan sweep must NOT run under a savepoint. - // - // `rollback_to(savepoint)` rewinds the roots + cache watermark but does NOT - // reset the structural recycle streams. The only path that COWs the freemap - // (mutating those streams) while a savepoint is open is the defrag orphan - // sweep. If the sweep ran under a savepoint, it could drain a committed-LIVE - // freemap page into `structural_superseded`; after `rollback_to` (which - // leaves the stream intact) + commit (which promotes it), the NEXT - // transaction would reuse that still-durably-referenced page as a COW target - // and overwrite it — silent durable freemap corruption. - // - // The fix guards `reclaim_freemap_orphans` with `savepoints.is_empty()`. - // This test reproduces the trigger end-to-end and asserts the committed - // freemap tree survives intact. Counterfactual: removing the guard makes the - // committed-tree-intact assertion (or the no-reuse-of-committed-page check) - // fail. - #[test] - fn orphan_sweep_skipped_under_savepoint_preserves_committed_freemap() { - let mut tm = fresh_manager(); - let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; - - // Build a real multi-page freemap with committed structural state so the - // committed tree has actual nodes to corrupt. After this the committed - // freemap root/depth describe a non-trivial tree. - let survivors = structural_churn(&mut tm, &big, 8, 4); - - // Snapshot the committed freemap's free-set and reachable node set BEFORE - // the savepoint episode. These are the ground truth the episode must not - // disturb. - let committed_root = tm.committed_roots.freemap_page; - let committed_depth = tm.committed_roots.freemap_depth; - assert_ne!( - committed_root, PAGE_ID_NONE, - "precondition: a committed freemap tree must exist" - ); - let (free_before, reachable_before): ( - std::collections::BTreeSet, - std::collections::HashSet, - ) = { - let mut cache = tm.cache.borrow_mut(); - let tree = FreeMapTree::from_roots(committed_root, committed_depth); - let reachable: std::collections::HashSet = tree - .reachable_pages(&mut cache) - .unwrap() - .into_iter() - .collect(); - // The set of currently-free ids, scanned over the allocation range. - let total = cache.next_page_id(); - let mut free = std::collections::BTreeSet::new(); - for id in 0..total { - if tree.is_free(&mut cache, id).unwrap() { - free.insert(id); - } - } - (free, reachable) - }; - - // Episode: open a transaction, take a savepoint, forge a freemap orphan, - // and invoke the sweep. With the guard the sweep is a no-op (returns 0) - // and touches NO structural stream; without the guard it would reclaim the - // forged orphan, COWing the committed freemap and draining the superseded - // live page into `structural_superseded`. - tm.begin().unwrap(); - tm.savepoint("sp").unwrap(); - let _orphan = tm.test_forge_freemap_orphan().unwrap(); - let reclaimed = tm.reclaim_freemap_orphans().unwrap(); - assert_eq!( - reclaimed, 0, - "the orphan sweep must be a no-op under an active savepoint (got {reclaimed})" - ); - // The streams the rollback_to does NOT reset must be untouched by the - // sweep, or the rollback leaves dangerous residue. - assert!( - tm.structural_superseded.is_empty(), - "sweep under savepoint leaked into structural_superseded: {:?}", - tm.structural_superseded - ); - - // Roll back to the savepoint (discards the forged page) and commit the - // now-empty transaction. With the guard this commit promotes nothing - // dangerous; without it, the committed-live page the sweep superseded is - // promoted into the reusable pool. - tm.rollback_to("sp").unwrap(); - tm.commit().unwrap(); - - // Next transaction does a freemap-COWing operation (delete a survivor, - // which marks its page free and COWs the freemap). If a committed-live - // freemap page had been promoted into the reuse pool, this is where it - // would be drawn as a COW target and OVERWRITTEN. - tm.begin().unwrap(); - tm.delete(survivors[0]).unwrap(); - tm.commit().unwrap(); - - // The committed freemap tree the ORIGINAL (pre-episode) commit described - // must still be readable and self-consistent: no node it referenced was - // overwritten. We re-open the ORIGINAL committed root/depth and confirm - // its reachable set and free-set are unchanged by everything above. - // (Deleting survivors[0] in the final txn only ADDS a free bit; it never - // removes one and never makes a previously-reachable node unreadable.) - let (free_after, reachable_after): ( - std::collections::BTreeSet, - std::collections::HashSet, - ) = { - let mut cache = tm.cache.borrow_mut(); - let tree = FreeMapTree::from_roots(committed_root, committed_depth); - let reachable: std::collections::HashSet = tree - .reachable_pages(&mut cache) - .unwrap() - .into_iter() - .collect(); - let total = cache.next_page_id(); - let mut free = std::collections::BTreeSet::new(); - for id in 0..total { - if tree.is_free(&mut cache, id).unwrap() { - free.insert(id); - } - } - (free, reachable) - }; - assert_eq!( - reachable_after, reachable_before, - "the original committed freemap tree's node set changed — a committed \ - freemap page was reused-and-overwritten (savepoint guard regression)" - ); - assert_eq!( - free_after, free_before, - "the original committed freemap tree's free-set changed — its on-disk \ - bitmap pages were overwritten by a COW into a reused committed page" - ); - } - - // A corrupt DEAD (non-reachable) page must NOT poison the orphan sweep. - // - // 2026-06-22 review decision ("skip unreadable dead pages"): the sweep scans - // every non-reachable page id to classify it as a freemap orphan. A page that - // is not in the live tree but fails to read because it is GARBAGE cannot be - // confirmed as an orphan, and its corruption is irrelevant (it is dead), so - // the sweep SKIPS it instead of propagating fatal. Contrast: a corrupt page - // REACHABLE from the live tree must still surface fatal via `reachable_pages` - // — that path is deliberately NOT weakened (asserted below). - #[test] - fn corrupt_dead_page_does_not_poison_orphan_sweep() { - let mut tm = fresh_manager(); - let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; - - // Build a committed multi-page freemap and a real orphan to reclaim, so - // the sweep has live work to do AND a corrupt dead page to step over. - let _survivors = structural_churn(&mut tm, &big, 8, 4); - - tm.begin().unwrap(); - let real_orphan = tm.test_forge_freemap_orphan().unwrap(); - let corrupt = tm.test_forge_corrupt_dead_page().unwrap(); - assert_ne!(real_orphan, corrupt); - - // The sweep must succeed (NOT poison) despite the corrupt dead page, and - // must still reclaim the legitimate orphan it can read. - let reclaimed = tm.reclaim_freemap_orphans().unwrap(); - assert!( - reclaimed >= 1, - "sweep must skip the corrupt dead page yet still reclaim the readable \ - orphan (reclaimed={reclaimed})" - ); - assert!( - !tm.is_poisoned(), - "a corrupt DEAD page must not poison the orphan sweep" - ); - tm.commit().unwrap(); - - // Contrast: the live-tree walk MUST still propagate fatal on a corrupt - // node reachable from the committed tree — `reachable_pages` is NOT - // weakened by the dead-page softening above. Corrupt a REAL live node on - // disk (flip its type byte to the wrong PageType but keep the checksum - // valid — a type-corruption reached via a live pointer) and confirm the - // walk surfaces `CorruptPage` for exactly that node. - let live_root = tm.committed_roots.freemap_page; - let live_depth = tm.committed_roots.freemap_depth; - { - let mut cache = tm.cache.borrow_mut(); - // Pick a non-root reachable node so the root's own type check passes - // and the failure happens during descent (the path the softening must - // NOT touch). If the tree is a single root (depth 0), the root IS the - // only node; corrupt it directly. - let tree = FreeMapTree::from_roots(live_root, live_depth); - let reachable: Vec = tree - .reachable_pages(&mut cache) - .unwrap() - .into_iter() - .collect(); - let victim = reachable - .iter() - .copied() - .find(|&id| id != live_root) - .unwrap_or(live_root); - // Wrong type byte, valid checksum: not a bit-flip, a position-type - // corruption that `check_type` rejects. Flip FreeMap<->FreeMapInterior. - { - let buf = cache.get_mut(victim).unwrap(); - let wrong = if buf[0] == crate::page::PageType::FreeMap as u8 { - crate::page::PageType::FreeMapInterior as u8 - } else { - crate::page::PageType::FreeMap as u8 - }; - buf[0] = wrong; - page::stamp_checksum(buf); - } - let err = tree.reachable_pages(&mut cache).unwrap_err(); - assert!( - matches!(err, ChiselError::CorruptPage { .. }), - "reachable_pages must surface fatal CorruptPage on a corrupt LIVE \ - node — the dead-page softening must not weaken the live walk \ - (got {err:?})" - ); - } - } - - // PROPERTY 3 — no lost/double free across reuse cycles. - // - // Churn for several commits, each freeing whole pages. After EACH commit: - // (a) every page freed that commit reads `is_free == true` via a fresh - // `FreeMapTree::from_roots(committed root, depth)` (no lost free); and - // (b) NO page id is simultaneously reachable in the LIVE freemap tree and - // present in `structural_reuse`-derived pool (`pending_structural_frees`) - // — a reuse-pool page MUST be dead (no double-allocation: the same page - // cannot be both a live tree node and a free structural target). - #[test] - fn structural_recycle_no_lost_or_double_free() { - let mut tm = fresh_manager(); - let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; - - // Seed a rotating population so each round both allocates and frees whole - // pages, keeping the freemap COWing every commit. - let mut live: Vec = Vec::new(); - tm.begin().unwrap(); - for _ in 0..10 { - live.push(tm.allocate(&big).unwrap()); - } - tm.commit().unwrap(); - - for round in 0..8u32 { - // Delete half, allocate a fresh half: whole-page frees every commit. - let to_delete: Vec = live.drain(..5).collect(); - tm.begin().unwrap(); - for h in &to_delete { - tm.delete(*h).unwrap(); - } - // Capture this commit's data frees BEFORE commit clears the vector. - let freed_this_commit: Vec = tm.txn_freed_pages.clone(); - for _ in 0..5 { - live.push(tm.allocate(&big).unwrap()); - } - // Re-snapshot: allocations may have reused some freed ids already, - // pulling them back out of the free set. The invariant we pin is on - // the pages STILL freed at commit time, so take the union of frees and - // exclude any id re-claimed as a live value this same transaction. - tm.commit().unwrap(); - - assert!( - !freed_this_commit.is_empty(), - "round {round}: deletes must free at least one whole page" - ); - - // (a) No lost free: every page this commit freed (and did not re-claim - // as a live value) reads free in the committed tree. - let live_set: std::collections::HashSet = live.iter().copied().collect(); - { - let mut cache = tm.cache.borrow_mut(); - let committed = FreeMapTree::from_roots( - tm.committed_roots.freemap_page, - tm.committed_roots.freemap_depth, - ); - for id in &freed_this_commit { - // A freed data page re-claimed as a live value this same - // transaction is correctly NOT free; skip those. - if live_set.contains(id) { - continue; - } - assert!( - committed.is_free(&mut cache, *id).unwrap(), - "round {round}: page {id} was freed this commit but is NOT free in \ - the committed freemap tree — a lost free" - ); - } - } - - // (b) No double-free: a page in the structural reuse pool must be DEAD, - // i.e. never simultaneously reachable in the live freemap tree. Walk - // the committed tree and intersect with the deferred recycle pool. - let reachable: std::collections::HashSet = { - let mut cache = tm.cache.borrow_mut(); - let committed = FreeMapTree::from_roots( - tm.committed_roots.freemap_page, - tm.committed_roots.freemap_depth, - ); - committed - .reachable_pages(&mut cache) - .unwrap() - .into_iter() - .collect() - }; - for id in &tm.pending_structural_frees { - assert!( - !reachable.contains(id), - "round {round}: freemap page {id} is in the structural reuse pool AND \ - still reachable in the live committed freemap tree — a reuse-pool page \ - must be dead (handing it out as a COW target would double-allocate it)" - ); - } - // A reuse-pool page must also not be marked free in the bitmap (the - // two reclamation channels are disjoint by design — a structural page - // rides the in-memory pool, never the bitmap). - { - let mut cache = tm.cache.borrow_mut(); - let committed = FreeMapTree::from_roots( - tm.committed_roots.freemap_page, - tm.committed_roots.freemap_depth, - ); - for id in &tm.pending_structural_frees { - assert!( - !committed.is_free(&mut cache, *id).unwrap(), - "round {round}: structural-reuse page {id} is ALSO marked free in the \ - bitmap — the two reclamation channels overlap, risking a double hand-out" - ); - } - } - } - } - - // The defrag orphan-sweep reclaims a freemap-typed page that a crash would - // have stranded: forge one (a checksum-valid FreeMapInterior unreferenced by - // the live tree and not marked free), sweep, and confirm it now reads free in - // the committed tree. This is the crash-recovery story for the in-memory - // structural recycle — without the sweep these pages leak permanently. - #[test] - fn reclaim_freemap_orphans_marks_lost_freemap_pages_free() { - let mut tm = fresh_manager(); - // Churn so a real multi-page freemap (leaf + spine) exists: overflow-sized - // values give each handle its own page, so deletes free whole pages. - let big: Vec = vec![0xCD; MAX_INLINE_VALUE + 32]; - tm.begin().unwrap(); - let mut hs = Vec::new(); - for _ in 0..40 { - hs.push(tm.allocate(&big).unwrap()); - } - tm.commit().unwrap(); - tm.begin().unwrap(); - for h in hs.iter().step_by(2) { - tm.delete(*h).unwrap(); - } - tm.commit().unwrap(); - - // Forge an orphan exactly as a crash leaves a lost recycle-pool page: - // extended, freemap-typed, unreferenced, not free. - let orphan = tm.test_forge_freemap_orphan().unwrap(); - - tm.begin().unwrap(); - let reclaimed = tm.reclaim_freemap_orphans().unwrap(); - assert!(reclaimed >= 1, "the forged orphan must be reclaimed"); - tm.commit().unwrap(); - - // The orphan now reads free in the committed tree (data-reusable bitmap - // space, disjoint from the structural recycle pool). - let mut cache = tm.cache.borrow_mut(); - let tree = FreeMapTree::from_roots( - tm.committed_roots.freemap_page, - tm.committed_roots.freemap_depth, - ); - assert!( - tree.is_free(&mut cache, orphan).unwrap(), - "reclaimed orphan {orphan} must read free in the committed freemap" - ); - } - - // The sweep's exclusion set is load-bearing: a page CURRENTLY in the live - // in-memory recycle pool (`structural_reuse`) is LIVE recycling state, not an - // orphan. Reclaiming it into the bitmap while it is also pool-reusable would - // double-hand-out the page. Seed the pool with a forged freemap-typed page - // (matching the orphan shape in every respect EXCEPT pool membership) and - // assert the sweep skips it and leaves the pool untouched. - #[test] - fn reclaim_freemap_orphans_excludes_live_recycle_pool() { - let mut tm = fresh_manager(); - // Establish a real freemap tree so the sweep does not early-exit on a - // PAGE_ID_NONE root. - let big: Vec = vec![0xCD; MAX_INLINE_VALUE + 32]; - tm.begin().unwrap(); - let h = tm.allocate(&big).unwrap(); - tm.commit().unwrap(); - tm.begin().unwrap(); - tm.delete(h).unwrap(); - tm.commit().unwrap(); - - // Forge a freemap-typed page that WOULD be flagged as an orphan, then put - // it in the live reuse pool so the exclusion set must spare it. - let pooled = tm.test_forge_freemap_orphan().unwrap(); - - tm.begin().unwrap(); - tm.structural_reuse.push(pooled); - let reclaimed = tm.reclaim_freemap_orphans().unwrap(); - assert_eq!( - reclaimed, 0, - "a page in the live recycle pool must NOT be reclaimed as an orphan" - ); - assert!( - tm.structural_reuse.contains(&pooled), - "the sweep must leave the live recycle pool untouched" - ); - tm.rollback().unwrap(); - } - - // Regression test for ISSUES.md I28. I19 introduced `CacheFull` as - // an **operational** error (documented as "commit or rollback to - // recover"), but `commit_inner` runs `persist_freemap` BEFORE - // `cache.flush()` — and `persist_freemap` itself calls - // `allocate_data_page`, which may trip `maybe_evict`'s ceiling - // check when every existing cache entry is dirty. Pre-fix the - // resulting `CacheFull` propagated out of commit_inner and - // commit()'s poison wrapper poisoned the manager unconditionally. - // The recovery advice ("commit to flush") became impossible to - // follow because commit itself failed. - // - // Post-fix: commit drains the cache BEFORE persist_freemap, so the - // cap is always reachable via eviction when persist_freemap - // itself allocates. CacheFull cannot arise on the commit path. - // - // Setup note: we deliberately use a small `max_pages` so the - // strict cap is cheap to saturate with a few allocations. - // spillway_max_bytes=0 keeps CacheFull reachable (no spillway - // escape hatch), matching the pre-spillway path this test exercises. - #[test] - fn commit_does_not_poison_when_cache_is_at_strict_cap() { - // I66 (ISSUES.md, 2026-05-22): TempDir for RAII cleanup — - // replaces the pre-I66 NamedTempFile + std::mem::forget(file) - // pattern that leaked the temp path on every test run. - let _dir = TempDir::new().unwrap(); - let db_path = _dir.path().join("test.chisel"); - let io = PageIo::open(&db_path, false).unwrap(); - // max_pages=16 — big enough for baseline operations (handle-table - // root + superblocks + freemap) to coexist, small enough that a - // handful of big allocations saturate the strict cap quickly. - // spillway_max_bytes=0 means CacheFull fires at max_pages itself. - let cache = PageCache::new( - io, - 16 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let mut tm = TransactionManager::create_new(cache, 2).unwrap(); - tm.begin().unwrap(); - tm.commit().unwrap(); - - // Seed one handle so the victim transaction can produce a - // non-empty `txn_freed_pages` via delete. Without any frees AND - // with `current_freemap == committed_freemap`, `persist_freemap` - // takes its early-exit path and never allocates — which would - // mean it also cannot trip CacheFull, and the test would fail - // to reproduce the bug. - let big: Vec = vec![0x99; MAX_INLINE_VALUE + 32]; - tm.begin().unwrap(); - let victim = tm.allocate(&big).unwrap(); - tm.commit().unwrap(); - - // Victim transaction: delete to populate txn_freed_pages, then - // allocate until the cache saturates at the strict cap. - // CacheFull from an allocate() is operational — we catch it and - // proceed to commit, which is what we actually want to stress. - tm.begin().unwrap(); - tm.delete(victim).unwrap(); - let mut saturated = false; - for _ in 0..200 { - match tm.allocate(&big) { - Ok(_) => continue, - Err(ChiselError::CacheFull { .. }) => { - saturated = true; - break; - } - Err(e) => panic!("unexpected error during cache-fill setup: {e:?}"), - } - } - assert!( - saturated, - "test precondition: cache did not reach CacheFull in 200 allocations" - ); - assert!( - !tm.is_poisoned(), - "precondition: CacheFull from allocate() is operational and must not poison" - ); - - // The actual I28 check. Pre-fix, `persist_freemap`'s internal - // `allocate_data_page` trips the ceiling and propagates - // CacheFull out of commit_inner; commit()'s poison wrapper - // then sets the poison flag. Post-fix commit drains first. - let result = tm.commit(); - assert!( - result.is_ok(), - "I28: commit over a saturated cache should succeed; got {result:?}" - ); - assert!( - !tm.is_poisoned(), - "I28: CacheFull during commit must not poison — it's operational by design" - ); - } - - // =================================================================== - // BUG#2 (2026-06-16 deepdive): forward/reverse tag-map atomic staging. - // - // `allocate_tagged` maintains two maps that must stay in lockstep: the - // FORWARD map (each chunk's `HandleEntry.tag`, in the handle table) and - // the REVERSE map (tag -> handles, in the membership index, powering - // `handles_with_tag`/delete-by-tag). Before the fix the two roots were - // installed sequentially, so a NON-FATAL `CacheFull`/`SpillwayFull` - // striking between them committed a half-update: - // * allocate: the forward map gained the tag but the reverse did not - // (a tagged chunk invisible to `handles_with_tag`); - // * delete: the tombstone landed but the reverse entry stayed (a - // stale member that later escalates to a FATAL CorruptPage when - // surfaced and acted upon). - // Because those errors are non-fatal they do NOT poison the manager, so - // the half-update survives to `commit()` and onto disk. - // - // Atomic staging computes BOTH candidate roots in a fallible prepare - // phase and installs them together in an infallible phase, so a mid-op - // failure is a no-op for the INSTALLED state (neither map changes, the - // handle id is not burned, inline slot bookkeeping is clean). There is a - // bounded freemap-reuse residue on the abnormal path — see the - // `abort_allocate_prepare` doc and the - // `aborted_tagged_allocate_with_freemap_reuse_is_consistent_and_rollback_reclaims` - // test. The `fail_next_membership_op` hook fires a simulated CacheFull at - // exactly the reverse-map step — the precise divergence window — so these - // are deterministic, not timing-dependent. - - #[test] - fn allocate_membership_failure_leaves_maps_consistent() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - // The id the about-to-fail allocate will (try to) use. - let ghost = tm.current_roots.next_handle; - - tm.fail_next_membership_op.set(true); - let err = tm.allocate_tagged(b"payload", 7).unwrap_err(); - assert!( - matches!(err, ChiselError::CacheFull { .. }), - "expected the injected CacheFull, got {err:?}" - ); - assert!(!tm.is_poisoned(), "a non-fatal CacheFull must not poison"); - - // The forward (tag) and reverse (membership) maps MUST agree. Pre-fix - // the forward map carried tag 7 for `ghost` while the reverse index - // did not — a committed-out-of-sync divergence. - let in_reverse = tm.handles_with_tag(7).unwrap().contains(&ghost); - let in_forward = matches!(tm.tag(ghost), Ok(7)); - assert_eq!( - in_forward, in_reverse, - "forward/reverse tag maps diverged after a failed tagged allocate" - ); - - // Atomic staging makes the failed allocate a no-op for the installed - // state: neither map changed and the handle id was not even burned. - assert!( - !in_forward, - "failed allocate must not install the forward entry" - ); - assert_eq!( - tm.current_roots.next_handle, ghost, - "failed allocate must not burn the handle id" - ); - - // ...and the no-op extends to the R1 packing bookkeeping: the inline - // value's data slot was released, so no phantom live-slot count or ghost - // insert cursor survives to skew later packing / defrag density / page - // reclamation. (Pre-fix this leaked `{page: 1}` and `Some(page)`.) - assert!( - tm.current_live_slots.is_empty(), - "failed allocate left a phantom live-slot count: {:?}", - tm.current_live_slots - ); - assert_eq!( - tm.insert_cursor, None, - "failed allocate left a ghost insert cursor" - ); - - // The manager is still fully usable: a disarmed retry reuses the same - // id and is consistent across BOTH maps, in-session and after commit. - let h = tm.allocate_tagged(b"payload", 7).unwrap(); - assert_eq!(h, ghost, "retry should reuse the un-burned handle id"); - assert_eq!(tm.tag(h).unwrap(), 7); - assert!(tm.handles_with_tag(7).unwrap().contains(&h)); - tm.commit().unwrap(); - assert!(tm.handles_with_tag(7).unwrap().contains(&h)); - // C1: no page reachable from committed_roots may be free in the - // committed freemap — pins that the dropped prepare freed-lists never - // queued a still-referenced page. - assert_no_reachable_page_is_free(&tm); - } - - // The FORWARD-step counterpart: a non-fatal CacheFull during the - // handle-table insert (the step that eagerly bumps the descent depth via - // HandleTable::grow, and on a fresh DB lazily materializes the root). The - // prepare-abort must restore BOTH the depth and the handle-table root - // pointer and release the inline slot — a true no-op. - #[test] - fn allocate_handle_table_failure_leaves_maps_consistent() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - // Fresh DB: the handle table has never been materialized. - let saved_root = tm.current_roots.handle_table_page; - assert_eq!(saved_root, PAGE_ID_NONE, "precondition: empty handle table"); - let ghost = tm.current_roots.next_handle; - - tm.fail_next_handle_table_op.set(true); - let err = tm.allocate_tagged(b"payload", 7).unwrap_err(); - assert!( - matches!(err, ChiselError::CacheFull { .. }), - "expected the injected CacheFull, got {err:?}" - ); - assert!(!tm.is_poisoned(), "a non-fatal CacheFull must not poison"); - - // Complete no-op: the lazily-created root is reverted (back to - // PAGE_ID_NONE), the handle id is not burned, neither map has content, - // and the R1 packing state is clean. - assert_eq!( - tm.current_roots.handle_table_page, saved_root, - "failed forward step left a lazily-materialized handle-table root installed" - ); - assert_eq!(tm.current_roots.next_handle, ghost); - assert!( - matches!(tm.tag(ghost), Err(ChiselError::InvalidHandle(_))), - "ghost handle after failed allocate_tagged must be InvalidHandle" - ); - assert!(tm.handles_with_tag(7).unwrap().is_empty()); - assert!( - tm.current_live_slots.is_empty(), - "failed forward step left a phantom live-slot count: {:?}", - tm.current_live_slots - ); - assert_eq!(tm.insert_cursor, None); - - // Disarmed retry succeeds and is consistent across both maps. - let h = tm.allocate_tagged(b"payload", 7).unwrap(); - assert_eq!(h, ghost, "retry should reuse the un-burned handle id"); - assert_eq!(tm.tag(h).unwrap(), 7); - assert!(tm.handles_with_tag(7).unwrap().contains(&h)); - tm.commit().unwrap(); - assert_no_reachable_page_is_free(&tm); - } - - #[test] - fn delete_membership_failure_leaves_maps_consistent() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let h = tm.allocate_tagged(b"payload", 7).unwrap(); - tm.commit().unwrap(); - - tm.begin().unwrap(); - tm.fail_next_membership_op.set(true); - let err = tm.delete(h).unwrap_err(); - assert!( - matches!(err, ChiselError::CacheFull { .. }), - "expected the injected CacheFull, got {err:?}" - ); - assert!(!tm.is_poisoned(), "a non-fatal CacheFull must not poison"); - - // Pre-fix the tombstone (forward) was installed but the reverse entry - // stayed, so `read(h)` failed while `handles_with_tag` still listed - // `h` — exactly the divergence that later escalates to CorruptPage. - let in_reverse = tm.handles_with_tag(7).unwrap().contains(&h); - let still_live = tm.read(h).is_ok(); - assert_eq!( - still_live, in_reverse, - "forward/reverse tag maps diverged after a failed tagged delete" - ); - // Atomic staging makes the failed delete a COMPLETE no-op. - assert!(still_live, "failed delete must not install the tombstone"); - assert!(in_reverse, "failed delete must not drop the reverse entry"); - - // A disarmed retry succeeds and removes `h` from BOTH maps. - tm.delete(h).unwrap(); - assert!( - matches!(tm.read(h), Err(ChiselError::InvalidHandle(_))), - "handle must be InvalidHandle after successful delete" - ); - assert!(!tm.handles_with_tag(7).unwrap().contains(&h)); - tm.commit().unwrap(); - assert!(!tm.handles_with_tag(7).unwrap().contains(&h)); - assert_no_reachable_page_is_free(&tm); - } - - // Delete's durability guard (delete is the more dangerous direction: a - // stale reverse member for a tombstoned handle later escalates to a FATAL - // CorruptPage). Fail the reverse-map step, COMMIT, then REOPEN: the failed - // delete must have committed NOTHING — `h` is still live AND still a tagged - // member on disk. Pre-fix the tombstone committed while the reverse member - // stayed, so the reopened DB would read `h` as deleted yet still list it. - #[test] - fn delete_membership_failure_survives_reopen_consistently() { - let dir = TempDir::new().unwrap(); - let path = dir.path().join("bug2-del.chisel"); - - let open = |create: bool| -> TransactionManager { - let io = PageIo::open(&path, false).unwrap(); - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - if create { - TransactionManager::create_new(cache, 2).unwrap() - } else { - TransactionManager::open_existing(cache).unwrap() - } - }; - - let h = { - let mut tm = open(true); - tm.begin().unwrap(); - let h = tm.allocate_tagged(b"payload", 9).unwrap(); - tm.commit().unwrap(); - - tm.begin().unwrap(); - tm.fail_next_membership_op.set(true); - assert!(matches!( - tm.delete(h).unwrap_err(), - ChiselError::CacheFull { .. } - )); - // Commit the post-failure state: pre-fix this durably tombstones the - // forward map while the reverse keeps `h`; post-fix it is a no-op. - tm.commit().unwrap(); - h - }; - - // Reopen: the forward (read) and reverse (handles_with_tag) views must - // agree, and since the delete was a no-op both must still see `h`. - let tm = open(false); - let still_live = tm.read(h).is_ok(); - let in_reverse = tm.handles_with_tag(9).unwrap().contains(&h); - assert_eq!( - still_live, in_reverse, - "reopened forward/reverse views diverged for the failed delete's handle" - ); - assert!( - still_live && in_reverse, - "a failed tagged delete must commit nothing: h should survive in both maps \ - (read ok = {still_live}, in reverse index = {in_reverse})" - ); - } - - // Pins the documented `delete_with_tag` error contract: a mid-pass failure - // returns Err (NO TagDropProgress — the dropped-this-pass set is not - // reported), is non-fatal/recoverable, and leaves a CONSISTENT partial - // state because each delete_inner is atomic (BUG#2 staging) — so exactly the - // members processed before the failure are gone from BOTH maps, and the - // partial drop is committable and resumable. - #[test] - fn delete_with_tag_mid_pass_failure_is_consistent_and_drops_progress() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let members = [ - tm.allocate_tagged(b"m0", 5).unwrap(), - tm.allocate_tagged(b"m1", 5).unwrap(), - tm.allocate_tagged(b"m2", 5).unwrap(), - ]; - tm.commit().unwrap(); - - tm.begin().unwrap(); - // Let the 1st delete in the pass commit, fail the 2nd's membership op. - tm.fail_membership_op_after.set(2); - let err = tm.delete_with_tag(5, 3).unwrap_err(); - assert!( - matches!(err, ChiselError::CacheFull { .. }), - "expected the injected CacheFull, got {err:?}" - ); - assert!( - !tm.is_poisoned(), - "a non-fatal mid-pass error must not poison" - ); - - // Exactly one member dropped before the failure (delete_inner is atomic, - // so the failed 2nd delete is a no-op). Don't assume index iteration - // order — count instead. - let live: Vec = members - .into_iter() - .filter(|&h| tm.read(h).is_ok()) - .collect(); - assert_eq!( - live.len(), - 2, - "exactly one member dropped before the failure" - ); - // Forward (read-ok set) and reverse (membership index) agree exactly. - let mut idx = tm.handles_with_tag(5).unwrap(); - let mut live_sorted = live.clone(); - idx.sort_unstable(); - live_sorted.sort_unstable(); - assert_eq!( - live_sorted, idx, - "forward/reverse maps consistent after a failed delete_with_tag pass" - ); - - // The consistent partial drop is committable... - tm.commit().unwrap(); - assert_eq!(tm.handles_with_tag(5).unwrap().len(), 2); - assert_no_reachable_page_is_free(&tm); - - // ...and the bounded loop finishes cleanly on a disarmed retry. - tm.begin().unwrap(); - let (_, complete) = tm.delete_with_tag(5, 3).unwrap(); - assert!(complete, "retry must drain the tag"); - tm.commit().unwrap(); - assert!(tm.handles_with_tag(5).unwrap().is_empty()); - assert_no_reachable_page_is_free(&tm); - } - - // CRITICAL durable-corruption regression (surfaced by the BUG#2 adversarial - // review): `update_inner` must not free the OLD value's pages before the new - // entry is durably installed. A non-fatal CacheFull at the new-value-write - // step — which pre-fix runs AFTER the old free — left the committed handle - // still pointing at pages already queued for reclamation, so commit freed a - // reachable page and a later reuse silently corrupted the live value. - #[test] - fn update_value_write_failure_does_not_free_old_value_pages() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - // Overflow-sized old value, so the whole chain is at stake. - let big = vec![0xABu8; MAX_INLINE_VALUE * 3]; - let h = tm.allocate(&big).unwrap(); - tm.commit().unwrap(); - assert_eq!(tm.read(h).unwrap(), big, "precondition: old value readable"); - - tm.begin().unwrap(); - tm.fail_next_update_value_write.set(true); - let err = tm.update(h, b"replacement").unwrap_err(); - assert!( - matches!(err, ChiselError::CacheFull { .. }), - "expected the injected CacheFull, got {err:?}" - ); - assert!(!tm.is_poisoned(), "a non-fatal CacheFull must not poison"); - - // The failed update is a no-op in-session: the old value is intact. - assert_eq!( - tm.read(h).unwrap(), - big, - "failed update lost the old value in-session" - ); - - // Commit the post-failure state, then assert C1: no page the committed - // handle still references may be free. Pre-fix the old overflow chain was - // queued into txn_freed_pages and freed here while `h` still points at it. - tm.commit().unwrap(); - assert_no_reachable_page_is_free(&tm); - assert_eq!( - tm.read(h).unwrap(), - big, - "failed update lost the old value after commit" - ); - - // End-to-end: churn allocations to force the freemap to hand out any - // wrongly-freed pages, then confirm the old value survived. Pre-fix the - // reachable-but-free chain pages would be reused and overwritten, turning - // read(h) into a CorruptPage / wrong bytes. - tm.begin().unwrap(); - for i in 0..40u8 { - tm.allocate(&[i; 64]).unwrap(); - } - tm.commit().unwrap(); - assert_eq!( - tm.read(h).unwrap(), - big, - "old value corrupted after its prematurely-freed pages were reused" - ); - } - - // Same guarantee for an INLINE old value (the Live old-release path). When - // the old page held only this value, the pre-fix code released it to the - // freemap before the new write, so a failed update committed a - // reachable-but-free data page just like the overflow case. - #[test] - fn update_value_write_failure_does_not_free_old_inline_page() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let old = b"inline-original-value"; - let h = tm.allocate(old).unwrap(); - tm.commit().unwrap(); - - tm.begin().unwrap(); - tm.fail_next_update_value_write.set(true); - assert!(matches!( - tm.update(h, b"replacement").unwrap_err(), - ChiselError::CacheFull { .. } - )); - assert!(!tm.is_poisoned()); - assert_eq!( - tm.read(h).unwrap(), - old, - "failed update lost the inline value" - ); - - tm.commit().unwrap(); - assert_no_reachable_page_is_free(&tm); - assert_eq!(tm.read(h).unwrap(), old); - - // Force reuse, then confirm the old inline value survived. - tm.begin().unwrap(); - for i in 0..40u8 { - tm.allocate(&[i; 64]).unwrap(); - } - tm.commit().unwrap(); - assert_eq!( - tm.read(h).unwrap(), - old, - "old inline value corrupted after a prematurely-freed page was reused" - ); - } - - // Highest-value coverage for the post-write prepare-unwind contract: the new - // value is ALREADY written when the handle-table install fails. The fix must - // (a) leave the OLD location referenced (never freed) and (b) release the - // just-written NEW inline slot so no phantom live-slot / ghost cursor - // survives. Driven via the shared fail_next_handle_table_op hook, which - // handle_table_insert_candidate honors for both allocate and update. (The - // old-overflow-walk failure exit runs the identical new-inline-slot release, - // so this test covers that contract too.) - #[test] - fn update_handle_table_failure_preserves_old_value_and_releases_new_slot() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let old = b"inline-original-value"; - let h = tm.allocate(old).unwrap(); - tm.commit().unwrap(); - - tm.begin().unwrap(); - tm.fail_next_handle_table_op.set(true); - let err = tm.update(h, b"small-new").unwrap_err(); - assert!( - matches!(err, ChiselError::CacheFull { .. }), - "expected the injected CacheFull, got {err:?}" - ); - assert!(!tm.is_poisoned()); - - // (a) The old value is untouched — the old location was never freed. - assert_eq!(tm.read(h).unwrap(), old, "failed update lost the old value"); - // (b) The new value's inline slot was released: the committed baseline - // had exactly one live slot (for `old`), and the failed update must - // leave precisely that — no phantom count for the abandoned new value. - assert_eq!( - tm.current_live_slots.values().sum::(), - 1, - "failed update left a phantom live-slot: {:?}", - tm.current_live_slots - ); - - // Durability: commit, assert C1, force reuse, re-read the old value. - tm.commit().unwrap(); - assert_no_reachable_page_is_free(&tm); - tm.begin().unwrap(); - for i in 0..40u8 { - tm.allocate(&[i; 64]).unwrap(); - } - tm.commit().unwrap(); - assert_eq!( - tm.read(h).unwrap(), - old, - "old value corrupted after reuse following a failed update" - ); - } - - // The headline durability guard: a failed tagged allocate, then COMMIT, - // then REOPEN from the last durable superblock. Pre-fix this persisted a - // forward-map ghost (a committed `HandleEntry.tag` with no reverse-index - // member); post-fix the failed allocate committed nothing. - #[test] - fn allocate_membership_failure_survives_reopen_consistently() { - let dir = TempDir::new().unwrap(); - let path = dir.path().join("bug2.chisel"); - - let open = |create: bool| -> TransactionManager { - let io = PageIo::open(&path, false).unwrap(); - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - if create { - TransactionManager::create_new(cache, 2).unwrap() - } else { - TransactionManager::open_existing(cache).unwrap() - } - }; - - let ghost = { - let mut tm = open(true); - tm.begin().unwrap(); - tm.commit().unwrap(); - - tm.begin().unwrap(); - let ghost = tm.current_roots.next_handle; - tm.fail_next_membership_op.set(true); - assert!(matches!( - tm.allocate_tagged(b"payload", 9).unwrap_err(), - ChiselError::CacheFull { .. } - )); - // Commit the post-failure state: pre-fix this makes the forward-map - // ghost durable; post-fix it commits a clean no-op. - tm.commit().unwrap(); - ghost - }; - - // Reopen and verify the FORWARD map carries no ghost tag-9 entry that - // the REVERSE map is missing. `handles_with_tag(9)` reads the reverse - // map (empty under both code paths); the discriminating check is the - // forward map via `tag(ghost)`. - let tm = open(false); - let forward = tm.tag(ghost); - assert!( - forward.is_err() || forward.as_ref().unwrap() != &9, - "reopened forward map has a ghost tag-9 entry (tag({ghost}) = {forward:?}) \ - with no matching reverse-index member — the maps committed out of sync" - ); - assert!( - tm.handles_with_tag(9).unwrap().is_empty(), - "tag 9 should have no committed members" - ); - } - - // When savepoints are absent (`reuse = true`), a failed tagged-allocate - // prepare may exercise the freemap REUSE path inside `cow_alloc` — the - // candidate COW clears free bits and advances the freemap tree BEFORE - // learning the membership step will fail. The abort restores the installed - // state (maps, handle id, inline slot) but intentionally leaves the - // freemap-reuse residue (bounded allocated-but-unreferenced pages), relying - // on rollback to discard them. This test seeds a committed freemap with free - // pages (so reuse fires), triggers the injection, and verifies: - // - // 1. The failed call returns the injected CacheFull. - // 2. The installed state is unchanged (both maps, next_handle, depth). - // 3. No C1 violation: every page reachable in the committed state is - // consistent after the aborted allocate. - // 4. rollback() returns the freemap to its committed free-set: a fresh - // `FreeMapTree::from_roots(committed_root, committed_depth)` reports - // the same free bits as before the aborted allocate. - #[test] - fn aborted_tagged_allocate_with_freemap_reuse_is_consistent_and_rollback_reclaims() { - let mut tm = fresh_manager(); - - // --- Phase 1: seed a committed freemap with free DATA pages --- - // - // Allocate several overflow-sized values (each occupies at least one - // whole page beyond the superblock/freemap spine), commit, then delete - // some and commit again. The second commit's persist_freemap marks those - // pages free, so the committed freemap now has reuse bits set. - let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; - let mut live_handles = Vec::new(); - tm.begin().unwrap(); - for _ in 0..6 { - live_handles.push(tm.allocate_tagged(&big, 5).unwrap()); - } - tm.commit().unwrap(); - - // Delete the first 3 to free their overflow pages into the committed - // freemap. Tag 5 still has 3 live members after this commit. - tm.begin().unwrap(); - for h in live_handles.drain(..3) { - tm.delete(h).unwrap(); - } - tm.commit().unwrap(); - assert_no_reachable_page_is_free(&tm); - - // Precondition: the committed freemap must have free bits so that the - // next `cow_alloc` exercises the reuse path (not just extend). - let pre_fm_root = tm.committed_roots.freemap_page; - let pre_fm_depth = tm.committed_roots.freemap_depth; - let pre_next_handle = tm.committed_roots.next_handle; - let pre_ht_root = tm.committed_roots.handle_table_page; - let pre_mi_root = tm.committed_roots.membership_index_page; - assert_ne!( - pre_fm_root, - crate::page::PAGE_ID_NONE, - "precondition: committed freemap must be non-empty (need free bits for reuse)" - ); - - // Snapshot the committed free set so we can compare after rollback. - let committed_free_before: std::collections::BTreeSet = { - let tree = FreeMapTree::from_roots(pre_fm_root, pre_fm_depth); - let mut cache = tm.cache.borrow_mut(); - let total = tm.committed_roots.total_pages; - (0..total) - .filter(|&id| tree.is_free(&mut cache, id).unwrap_or(false)) - .collect() - }; - assert!( - !committed_free_before.is_empty(), - "precondition: at least one free page in committed freemap for reuse path" - ); - - // --- Phase 2: begin a new transaction (no savepoints → reuse enabled) - // and inject the membership failure --- - tm.begin().unwrap(); - - // Savepoints must be empty so reuse is live. - assert!( - tm.savepoints.is_empty(), - "precondition: no savepoints, so freemap reuse is enabled" - ); - - let ghost = tm.current_roots.next_handle; - let saved_ht_depth = tm.handle_table.depth(); - - // Fire the injection: `cow_alloc` inside `handle_table_insert_candidate` - // or `membership_insert_candidate` will draw from the committed freemap - // before the prepare fails. - tm.fail_next_membership_op.set(true); - let err = tm.allocate_tagged(&big, 5).unwrap_err(); - assert!( - matches!(err, ChiselError::CacheFull { .. }), - "expected injected CacheFull, got {err:?}" - ); - assert!(!tm.is_poisoned(), "non-fatal CacheFull must not poison"); - - // --- Assertion 1: installed state is unchanged --- - assert_eq!( - tm.current_roots.next_handle, ghost, - "aborted allocate must not consume the handle id" - ); - assert_eq!( - tm.current_roots.handle_table_page, pre_ht_root, - "aborted allocate must not install a new handle-table root" - ); - assert_eq!( - tm.current_roots.membership_index_page, pre_mi_root, - "aborted allocate must not install a new membership-index root" - ); - assert_eq!( - tm.handle_table.depth(), - saved_ht_depth, - "aborted allocate must restore the eagerly-bumped handle-table depth" - ); - // Neither map has `ghost`. - assert!( - matches!(tm.tag(ghost), Err(ChiselError::InvalidHandle(_))), - "ghost handle must not appear in forward map" - ); - assert!( - !tm.handles_with_tag(5).unwrap().contains(&ghost), - "ghost handle must not appear in reverse (membership) map" - ); - // next_handle was not advanced. - assert_eq!( - tm.current_roots.next_handle, pre_next_handle, - "next_handle must equal the committed baseline (not burned)" - ); - - // --- Assertion 2: rollback reclaims the freemap residue --- - tm.rollback().unwrap(); - - // After rollback, committed_roots is unchanged (rollback restores - // current_roots to committed_roots, which did not change mid-transaction). - assert_eq!(tm.committed_roots.freemap_page, pre_fm_root); - assert_eq!(tm.committed_roots.freemap_depth, pre_fm_depth); - - // The free set as seen through the committed freemap tree must equal - // the pre-allocate snapshot: rollback's discard_all_dirty + truncate - // restored the freemap's bit pattern to the committed state. - let committed_free_after: std::collections::BTreeSet = { - let tree = FreeMapTree::from_roots( - tm.committed_roots.freemap_page, - tm.committed_roots.freemap_depth, - ); - let mut cache = tm.cache.borrow_mut(); - let total = tm.committed_roots.total_pages; - (0..total) - .filter(|&id| tree.is_free(&mut cache, id).unwrap_or(false)) - .collect() - }; - assert_eq!( - committed_free_before, committed_free_after, - "rollback must restore the committed freemap free-set: \ - before={committed_free_before:?} after={committed_free_after:?}" - ); - - // --- Assertion 3: C1 holds after rollback (no page reachable from - // committed_roots is marked free) --- - assert_no_reachable_page_is_free(&tm); - - // --- Assertion 4: the manager is still fully operational --- - // A new transaction can retry the allocation and succeed. - tm.begin().unwrap(); - let h = tm.allocate_tagged(&big, 5).unwrap(); - assert_eq!(h, ghost, "retry must reuse the un-burned handle id"); - assert_eq!(tm.tag(h).unwrap(), 5); - assert!(tm.handles_with_tag(5).unwrap().contains(&h)); - tm.commit().unwrap(); - assert_no_reachable_page_is_free(&tm); - } - - // I29: the open-time format-version gate compares MAJOR only, not - // the full u32. Same-major files (regardless of minor) open cleanly; - // different-major files fail fast with UnsupportedFormatVersion. - // This encodes the "sacred within a major version" promise from the - // README: any file written by an N.x binary is readable by every - // other N.x binary, because minor bumps can only add fields in the - // superblock's reserved region (they never break backward reads). - // - // We exercise both halves in one test because they share setup - // (fresh-DB file → patch slots → reopen). Patching both superblock - // slots in lockstep matters because Superblock::select picks the - // highest-counter valid slot; if only slot 0 were patched, slot 1 - // (with the unmodified version) would win on some commits. - #[test] - fn format_version_gate_is_major_only() { - let file = NamedTempFile::new().unwrap(); - let path = file.path().to_path_buf(); - - // Step 0: create a fresh database so there's something on disk - // to patch. The default superblock_count of 2 means we need to - // patch pages 0 AND 1. - { - let io = PageIo::open(&path, false).unwrap(); - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let _ = TransactionManager::create_new(cache, 2).unwrap(); - // drop() releases the flock so the test can read+write the - // file directly below. - } - - // Helper: patch every superblock slot to the given packed - // format_version and re-stamp the trailing checksum so the - // deserialize path still accepts it as a valid slot. - let patch_all_slots = |version: u32, slot_count: usize| { - let mut bytes = std::fs::read(&path).unwrap(); - for slot in 0..slot_count { - let offset = slot * PAGE_SIZE; - bytes[offset + 4..offset + 8].copy_from_slice(&version.to_le_bytes()); - let page_arr: &mut [u8; PAGE_SIZE] = - (&mut bytes[offset..offset + PAGE_SIZE]).try_into().unwrap(); - page::stamp_checksum(page_arr); - } - std::fs::write(&path, &bytes).unwrap(); - }; - - // Case 1: patch both slots to (FORMAT_MAJOR_VERSION, +42). A minor - // bump within the same major must open cleanly — this is the - // whole point of the packed scheme. Pre-fix (exact-match gate) - // this case rejected with UnsupportedFormatVersion. - let minor_bump = page::pack_format_version( - page::FORMAT_MAJOR_VERSION, - page::FORMAT_MINOR_VERSION.wrapping_add(42), - ); - patch_all_slots(minor_bump, 2); - { - let io = PageIo::open(&path, false).unwrap(); - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let tm = TransactionManager::open_existing(cache); - assert!( - tm.is_ok(), - "same-major / different-minor file should open cleanly; got {:?}", - tm.err() - ); - } - - // Case 2: patch to (FORMAT_MAJOR_VERSION + 1, 0). A major bump - // is a real format break and must be refused regardless of minor. - let major_bump = page::pack_format_version(page::FORMAT_MAJOR_VERSION.wrapping_add(1), 0); - patch_all_slots(major_bump, 2); - { - let io = PageIo::open(&path, false).unwrap(); - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - match TransactionManager::open_existing(cache) { - Err(ChiselError::UnsupportedFormatVersion { .. }) => {} - Err(e) => panic!("expected UnsupportedFormatVersion, got {e:?}"), - Ok(_) => panic!("expected UnsupportedFormatVersion, got Ok"), - } - } - } - - // I29 write-gate: a file whose MINOR exceeds this binary's opens READ-ONLY, - // not rejected — within a MAJOR every layout change is additive, so reads - // are safe, but writing would drop fields this binary can't see. A - // same-or-older minor file opens read-write as normal. - #[test] - fn file_minor_newer_than_binary_is_forced_read_only() { - let file = NamedTempFile::new().unwrap(); - let path = file.path().to_path_buf(); - - // Fresh DB so there are 2 superblock slots (pages 0 and 1) to patch. - { - let io = PageIo::open(&path, false).unwrap(); - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let _ = TransactionManager::create_new(cache, 2).unwrap(); - } - - // Patch every slot to (current MAJOR, MINOR + 1) and re-stamp checksums. - let newer_minor = - page::pack_format_version(page::FORMAT_MAJOR_VERSION, page::FORMAT_MINOR_VERSION + 1); - let mut bytes = std::fs::read(&path).unwrap(); - for slot in 0..2 { - let offset = slot * PAGE_SIZE; - bytes[offset + 4..offset + 8].copy_from_slice(&newer_minor.to_le_bytes()); - let page_arr: &mut [u8; PAGE_SIZE] = - (&mut bytes[offset..offset + PAGE_SIZE]).try_into().unwrap(); - page::stamp_checksum(page_arr); - } - std::fs::write(&path, &bytes).unwrap(); - - // Reopen read-WRITE; the gate must force read-only, so begin() fails. - let io = PageIo::open(&path, false).unwrap(); - let cache = PageCache::new( - io, - 1024 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let mut tm = TransactionManager::open_existing(cache) - .expect("a newer-minor file must still OPEN (reads are additive-safe)"); - assert!( - matches!(tm.begin(), Err(ChiselError::ReadOnlyMode)), - "a newer-minor file must be forced read-only" - ); - } - - #[test] - fn allocate_tagged_then_tag_and_handles_with_tag() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let h = tm.allocate_tagged(b"row", 42).unwrap(); - let u = tm.allocate(b"untagged").unwrap(); - tm.commit().unwrap(); - assert_eq!(tm.tag(h).unwrap(), 42); - assert_eq!(tm.tag(u).unwrap(), 0); - assert_eq!(tm.handles_with_tag(42).unwrap(), vec![h]); - assert_eq!(tm.handles_with_tag(99).unwrap(), Vec::::new()); - } - - #[test] - fn handles_with_tag_accumulates_multiple_handles() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let a = tm.allocate_tagged(b"a", 42).unwrap(); - let b = tm.allocate_tagged(b"b", 42).unwrap(); - let c = tm.allocate_tagged(b"c", 42).unwrap(); - tm.commit().unwrap(); - // The reverse index accumulates members; it must not overwrite. - let mut got = tm.handles_with_tag(42).unwrap(); - got.sort(); - let mut want = vec![a, b, c]; - want.sort(); - assert_eq!(got, want); - assert_eq!(tm.tag(a).unwrap(), 42); - assert_eq!(tm.tag(c).unwrap(), 42); - } - - #[test] - fn allocate_tagged_overflow_value_preserves_tag() { - let mut tm = fresh_manager(); - // A value larger than MAX_INLINE_VALUE takes the overflow path in - // allocate_inner; the tag must be stored on the Overflow HandleEntry, - // readable via tag(), and indexed for handles_with_tag(). - let big = vec![0xABu8; MAX_INLINE_VALUE + 100]; - tm.begin().unwrap(); - let h = tm.allocate_tagged(&big, 77).unwrap(); - tm.commit().unwrap(); - assert_eq!(tm.tag(h).unwrap(), 77); - assert_eq!(tm.handles_with_tag(77).unwrap(), vec![h]); - assert_eq!(tm.read(h).unwrap(), big); - } - - #[test] - fn update_preserves_immutable_tag() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let h = tm.allocate_tagged(b"v1", 42).unwrap(); - tm.commit().unwrap(); - tm.begin().unwrap(); - tm.update(h, b"v2").unwrap(); - tm.commit().unwrap(); - // Tags are immutable: changing the value must NOT change the tag, and the - // membership index must still list the handle under its original tag. - assert_eq!(tm.tag(h).unwrap(), 42); - assert_eq!(tm.handles_with_tag(42).unwrap(), vec![h]); - } - - #[test] - fn delete_removes_tagged_chunk_from_index() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let h = tm.allocate_tagged(b"row", 7).unwrap(); - tm.commit().unwrap(); - assert_eq!(tm.handles_with_tag(7).unwrap(), vec![h]); - tm.begin().unwrap(); - tm.delete(h).unwrap(); - tm.commit().unwrap(); - // The tag's last member is gone -> handles_with_tag is empty. - assert_eq!(tm.handles_with_tag(7).unwrap(), Vec::::new()); - } - - #[test] - fn delete_tagged_rejects_wrong_tag() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let h = tm.allocate_tagged(b"row", 5).unwrap(); - // Wrong tag: error, nothing deleted, index intact. - let err = tm.delete_tagged(h, 6).unwrap_err(); - assert!( - matches!(err, ChiselError::TagMismatch { handle, expected: 6, actual: 5 } if handle == h) - ); - assert_eq!(tm.handles_with_tag(5).unwrap(), vec![h]); - // TagMismatch is operational, NOT fatal: a wrong tag must leave the - // manager usable (guards against a future edit moving it into is_fatal). - assert!(!tm.is_poisoned()); - // Right tag: deletes (and self-maintains the index via delete_inner). - tm.delete_tagged(h, 5).unwrap(); - assert_eq!(tm.handles_with_tag(5).unwrap(), Vec::::new()); - tm.commit().unwrap(); - } - - #[test] - fn delete_one_of_two_tagged_keeps_the_other() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let a = tm.allocate_tagged(b"a", 7).unwrap(); - let b = tm.allocate_tagged(b"b", 7).unwrap(); - tm.commit().unwrap(); - tm.begin().unwrap(); - tm.delete(a).unwrap(); - tm.commit().unwrap(); - // Only `a` is removed from the reverse index; `b` survives under tag 7. - assert_eq!(tm.handles_with_tag(7).unwrap(), vec![b]); - assert_eq!(tm.tag(b).unwrap(), 7); - } - - // ── Migrated 2026-05-22 from tests/transactions.rs (I35 reshape) ── - // - // Exercises TransactionManager::open_existing end-to-end: a value - // written through one TransactionManager survives a drop + reopen - // on the same path. The other tests in tests/transactions.rs use - // only the public Chisel API and stay in tests/. - #[test] - fn reopen_preserves_committed_data() { - let file = NamedTempFile::new().unwrap(); - let path = file.path().to_owned(); - let handle; - { - let io = PageIo::open(&path, false).unwrap(); - let cache = PageCache::new( - io, - 64 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let mut txm = TransactionManager::create_new(cache, 2).unwrap(); - txm.begin().unwrap(); - handle = txm.allocate(b"persistent").unwrap(); - txm.commit().unwrap(); - } - { - let io = PageIo::open(&path, false).unwrap(); - let cache = PageCache::new( - io, - 64 * PAGE_SIZE as u64, - 0, - crate::DrainInsertion::LruTail, - crate::SpillwayLocation::InMemory, - ); - let txm = TransactionManager::open_existing(cache).unwrap(); - let data = txm.read(handle).unwrap(); - assert_eq!(data, b"persistent"); - } - } - - #[test] - fn delete_with_tag_drops_in_bounded_batches() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let mut hs = Vec::new(); - for i in 0..10u64 { - hs.push(tm.allocate_tagged(format!("row{i}").as_bytes(), 3).unwrap()); - } - tm.commit().unwrap(); - tm.begin().unwrap(); - let (d1, c1) = tm.delete_with_tag(3, 4).unwrap(); - assert_eq!(d1.len(), 4); - assert!(!c1); - let (d2, c2) = tm.delete_with_tag(3, 100).unwrap(); - assert_eq!(d2.len(), 6); - assert!(c2); - tm.commit().unwrap(); - assert_eq!(tm.handles_with_tag(3).unwrap(), Vec::::new()); - // The chunks themselves are gone too. - for h in hs { - assert!( - matches!(tm.read(h), Err(ChiselError::InvalidHandle(_))), - "handle {h} must be InvalidHandle after delete_with_tag" - ); - } - } - - #[test] - fn delete_with_tag_exact_max_reports_complete() { - // Boundary: exactly `max` members remain, so the max+1 enumeration - // returns exactly `max` and `complete = max <= max` must be TRUE. This - // is the <= vs < edge: a `<` here would falsely report incomplete and - // cost the caller an extra empty pass. (The other delete_with_tag test - // only exercises len > max and len < max, never len == max.) - let mut tm = fresh_manager(); - tm.begin().unwrap(); - for i in 0..5u64 { - tm.allocate_tagged(format!("r{i}").as_bytes(), 8).unwrap(); - } - tm.commit().unwrap(); - tm.begin().unwrap(); - let (deleted, complete) = tm.delete_with_tag(8, 5).unwrap(); - assert_eq!(deleted.len(), 5); - assert!( - complete, - "deleting exactly all members in one max-sized pass must report complete" - ); - tm.commit().unwrap(); - assert_eq!(tm.handles_with_tag(8).unwrap(), Vec::::new()); - } - - #[test] - fn handle_table_depth_restored_after_rolled_back_grow() { - // I99 regression: a rolled-back handle-table grow must not leave the - // in-memory depth too deep. A leaf holds ENTRIES_PER_LEAF (510) ids - // 0..=509; handle 0 is the reserved "no handle" sentinel, so allocation - // starts at id 1. Allocating ids 1..=509 (509 handles) stays at depth 0, - // and the next handle (id 510, where `id >= cap` triggers the grow) goes - // to depth 1. Roll that back, then a committed handle must still read (it - // returned InvalidHandle before the fix). - let mut tm = fresh_manager(); - tm.begin().unwrap(); - for i in 0..509u64 { - tm.allocate(format!("v{i}").as_bytes()).unwrap(); - } - tm.commit().unwrap(); - let baseline = tm.read(5).unwrap(); - tm.begin().unwrap(); - tm.allocate(b"grow").unwrap(); - tm.rollback().unwrap(); - assert_eq!( - tm.read(5).unwrap(), - baseline, - "committed handle lost after rolled-back grow" - ); - // New inserts still work (depth consistent for the next transaction). - tm.begin().unwrap(); - let h = tm.allocate(b"after").unwrap(); - tm.commit().unwrap(); - assert_eq!(tm.read(h).unwrap(), b"after"); - } - - #[test] - fn handle_table_depth_restored_after_rollback_to_savepoint() { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - // Handle 0 is the reserved "no handle" sentinel, so allocation starts at - // id 1: ids 1..=509 (509 handles) stay at depth 0, and the "grow" below - // (id 510, where `id >= ENTRIES_PER_LEAF` triggers the grow) is the one - // that crosses to depth 1, past the savepoint. Handle h holds "v{h-1}". - for i in 0..509u64 { - tm.allocate(format!("v{i}").as_bytes()).unwrap(); - } - tm.savepoint("sp").unwrap(); - tm.allocate(b"grow").unwrap(); // grows depth 0 -> 1 past the savepoint - tm.rollback_to("sp").unwrap(); - // A handle present at the savepoint must still read within the active txn. - // Handle 5 was the i=4 allocation, so it holds "v4". - assert_eq!(tm.read(5).unwrap(), b"v4"); - tm.commit().unwrap(); - assert_eq!(tm.read(5).unwrap(), b"v4"); - } - - #[test] - fn commit_fsync_failure_poisons_at_each_of_the_three_fsyncs() { - // I112: commit performs THREE fsyncs (pre-drain, data-flush, superblock). - // A real IoError at ANY of them must surface as IoError AND poison the - // manager. The FailFsync countdown targets each in turn. A small inline - // value keeps commit to exactly three fsyncs (no spillway: fresh_manager - // sets spillway_max_bytes=0). - for nth in 0..3u32 { - let mut tm = fresh_manager(); - tm.begin().unwrap(); - tm.allocate(b"v").unwrap(); - tm.cache.borrow().io().arm_fault(Fault::FailFsync(nth)); - let result = tm.commit(); - assert!( - matches!(result, Err(ChiselError::IoError(_))), - "commit fsync #{} failure must surface IoError, got {result:?}", - nth + 1 - ); - assert!(tm.is_poisoned(), "fsync #{} failure must poison", nth + 1); - assert!( - matches!(tm.read(0), Err(ChiselError::Poisoned)), - "a poisoned manager rejects all further ops" - ); - } - } - - #[test] - fn commit_write_failure_poisons() { - // I112: a real write_page IoError during commit must surface and poison. - // Target the value's own data page, which is written during commit flush. - let mut tm = fresh_manager(); - tm.begin().unwrap(); - let h = tm.allocate(b"v").unwrap(); - let pid = tm - .handle_live_page_id(h) - .unwrap() - .expect("allocated value has a live data page"); - tm.cache.borrow().io().arm_fault(Fault::FailWritePage(pid)); - let result = tm.commit(); - assert!( - matches!(result, Err(ChiselError::IoError(_))), - "commit write failure must surface IoError, got {result:?}" - ); - assert!(tm.is_poisoned(), "write failure during commit must poison"); - } -} diff --git a/src/transaction/config.rs b/src/transaction/config.rs new file mode 100644 index 0000000..849a55f --- /dev/null +++ b/src/transaction/config.rs @@ -0,0 +1,35 @@ +//! transaction::config — runtime knobs: cache / spillway byte caps and +//! the drain-insertion policy. Split out of `transaction.rs` verbatim; see +//! the parent module for the type and fields. + +use super::*; + +impl TransactionManager { + pub fn set_cache_max_bytes(&mut self, bytes: u64) -> Result<()> { + self.check_alive()?; + if self.active_txn { + return Err(ChiselError::TransactionInProgress); + } + self.cache.borrow_mut().set_cache_max_bytes(bytes) + } + + pub fn set_spillway_max_bytes(&mut self, bytes: u64) -> Result<()> { + self.check_alive()?; + if self.active_txn { + return Err(ChiselError::TransactionInProgress); + } + self.cache.borrow_mut().set_spillway_max_bytes(bytes) + } + + pub fn set_drain_insertion(&mut self, policy: crate::DrainInsertion) -> Result<()> { + self.check_alive()?; + if self.active_txn { + return Err(ChiselError::TransactionInProgress); + } + // I40: PageCache::set_drain_insertion is now infallible (returns + // `()`). The poison + active-txn checks above are the real + // failure modes; we promote PageCache's `()` to `Ok(())` here. + self.cache.borrow_mut().set_drain_insertion(policy); + Ok(()) + } +} diff --git a/src/transaction/fault.rs b/src/transaction/fault.rs new file mode 100644 index 0000000..5eec3b7 --- /dev/null +++ b/src/transaction/fault.rs @@ -0,0 +1,30 @@ +//! Test-only fault injection consolidated off the production TransactionManager +//! (review 2026-06-22 SMELL #4). Each Cell arms a one-shot or countdown failure +//! at a precise commit-protocol divergence window; see the BUG#2 staging tests. +//! +//! - `fail_next_membership_op`: the NEXT reverse-map (membership-index) update in +//! `allocate_inner`/`delete_inner` returns a non-fatal `CacheFull` BEFORE touching +//! the index, simulating a mid-operation resource-exhaustion strike at the +//! forward/reverse divergence window. Mirrors `force_poison_for_test` — no +//! production code. +//! - `fail_next_handle_table_op`: companion for the FORWARD step — the next +//! `allocate_inner` handle-table insert returns a non-fatal `CacheFull`, exercising +//! the prepare-abort/unwind path of the step carrying the eager depth bump +//! (HandleTable::grow). +//! - `fail_next_update_value_write`: for `update_inner` — the next update returns a +//! non-fatal `CacheFull` at the NEW-value-write step (the first fallible step with +//! the fix; pre-fix it landed AFTER the old location was freed), proving the old +//! value is not prematurely freed before the new entry installs. +//! - `fail_membership_op_after`: countdown variant of `fail_next_membership_op` for +//! multi-delete passes (e.g. delete_with_tag) — when set to K, the Kth subsequent +//! membership-index op fails (the first K-1 succeed), so a test can fail a LATER +//! delete in a loop while earlier deletes commit first. 0 disables. +use std::cell::Cell; + +#[derive(Default)] +pub(super) struct FaultInjector { + pub fail_next_membership_op: Cell, + pub fail_next_handle_table_op: Cell, + pub fail_next_update_value_write: Cell, + pub fail_membership_op_after: Cell, +} diff --git a/src/transaction/freemap.rs b/src/transaction/freemap.rs new file mode 100644 index 0000000..5eb8538 --- /dev/null +++ b/src/transaction/freemap.rs @@ -0,0 +1,479 @@ +//! transaction::freemap — freemap staging and structural-page recycling: +//! the freemap-aware COW allocator (`cow_alloc` / `structural_extend`), +//! `persist_freemap`, orphan reclamation, and the data-page / handle-table +//! COW allocation paths. Split out of `transaction.rs` verbatim; see the +//! parent module for the type and fields. + +use super::*; + +/// Freemap-aware page allocator shared by data-page allocation and the +/// handle-table / membership-index COW paths. +/// +/// When `reuse_enabled`, it asks the freemap `tree` for the lowest free id +/// at/above `*hint` (clearing its bit via a COW so it cannot be handed out +/// twice), falling back to extending the file via `PageCache::new_page`. +/// `reuse_enabled` is false while savepoints are active (R2: savepoint scopes +/// disable freemap reuse to keep `rollback_to` semantics simple) — matching the +/// historical `allocate_data_page` behavior, which this also routes through. +/// +/// The tree's own COW of the claimed leaf supersedes pages, which the caller +/// drains from `tree.pending_superseded` into `txn_freed_pages` after the call. +/// +/// LAZY-CREATE GUARD: a fresh database has `tree.root == PAGE_ID_NONE` (no tree +/// materialized yet). `PAGE_ID_NONE` is `u64::MAX`, NOT the tree's internal +/// zero-child sentinel, so `allocate_first` would try to read page u64::MAX and +/// error rather than reporting "nothing free". We short-circuit that here: a +/// None-root tree holds nothing reusable, so we fall straight through to +/// `new_page`. The tree is first materialized when a page is *freed* (see +/// persist_freemap), never on the allocation side. +/// +/// Pages freed during the CURRENT transaction live in `txn_freed_pages` and are +/// NOT in the committed tree until commit, so `allocate_first` can never hand +/// back a page still referenced by the live tree (the I18 invariant). Routing +/// handle-table and membership COW allocation through here — rather than the +/// monotonic `new_page` — is what lets those structures reach a bounded +/// steady-state page count instead of leaking one page per mutation. +pub(super) fn cow_alloc( + cache: &mut PageCache, + tree: &mut FreeMapTree, + hint: &mut u64, + structural_reuse: &mut Vec, + reuse_enabled: bool, +) -> Result { + if reuse_enabled && tree.root != PAGE_ID_NONE { + // `allocate_first` claims a free DATA page (clearing its bit), which COWs + // the freemap leaf. That leaf COW's structural `extend` reuses a dead + // freemap page from `structural_reuse` before extending the file — what + // keeps the freemap from marching the file upward one page per commit. + let mut extend = |c: &mut PageCache| structural_extend(c, structural_reuse); + if let Some(id) = tree.allocate_first(cache, hint, &mut extend)? { + cache.claim_page(id)?; + return Ok(id); + } + } + cache.new_page() +} + +// Verification hook (tests only): every page id drawn from `structural_reuse` +// as a freemap-COW target is recorded here, so the recycle pin-tests can assert +// the one-commit defer (a reused id was promoted by a PRIOR commit, never one +// this transaction itself superseded). A thread-local keeps the production +// `structural_extend` signature and both inline pop sites untouched; the +// recording calls are `#[cfg(test)]` no-ops in release builds. The single-writer +// model means at most one manager drives this per thread at a time. +#[cfg(test)] +thread_local! { + static STRUCTURAL_REUSE_LOG: RefCell> = const { RefCell::new(Vec::new()) }; +} + +#[cfg(test)] +fn record_structural_reuse(id: u64) { + STRUCTURAL_REUSE_LOG.with(|log| log.borrow_mut().push(id)); +} + +/// Drain and return every structural-reuse pop recorded since the last drain. +#[cfg(test)] +pub(super) fn take_structural_reuse_log() -> Vec { + STRUCTURAL_REUSE_LOG.with(|log| std::mem::take(&mut *log.borrow_mut())) +} + +/// Structural-page allocator for the freemap tree's COW: reuse a dead freemap +/// page (deferred from a prior commit, now safe to overwrite) before extending +/// the file. NEVER draws from the freemap's own free bits — that would re-COW a +/// leaf and recurse — preserving the extend-only termination guarantee while +/// bounding steady-state growth. `claim_page` evicts any stale cache entry for +/// the reused id before the COW writes its fresh contents. +fn structural_extend(cache: &mut PageCache, structural_reuse: &mut Vec) -> Result { + if let Some(id) = structural_reuse.pop() { + #[cfg(test)] + record_structural_reuse(id); + cache.claim_page(id)?; + Ok(id) + } else { + cache.new_page() + } +} + +impl TransactionManager { + // --- Watermark-based rollback (ISSUES.md I3 + I7) --- + // + // `PageCache::new_page()` hands out monotonically increasing ids, so + // every page allocated during a transaction has an id strictly greater + // than or equal to the `next_page_id` watermark captured at begin() / + // savepoint() time. `PageCache::truncate(watermark)` drops every cache + // entry AND truncates the file to `watermark` pages, cleanly discarding + // every transaction-allocated page without a per-page tracking list. + // + // This supersedes an earlier per-page `txn_dirty_pages` vector — the + // list was a weaker mechanism (I7 showed it missed intermediate COW + // pages and overflow allocations) and a redundant one once the + // watermark invariant was in place. See memory + // project_chisel_i3_watermark_rollback for the reasoning. + // + // Savepoints capture `cache.next_page_id()` at creation time (see the + // `watermark` field on Savepoint) so `rollback_to(name)` can truncate + // to that specific watermark — discarding every page allocated after + // the savepoint while preserving those allocated before it. + + /// Snapshot the current `next_page_id` watermark. Cheap — one read + /// through the RefCell. + pub(super) fn cache_watermark(&self) -> u64 { + self.cache.borrow().next_page_id() + } + + // --- Freemap-aware page allocation (ISSUES.md R2) --- + // + // `allocate_data_page` is the single entry point for allocating a + // fresh data page during a transaction. It first tries to reuse an + // id from `current_freemap` and falls back to extending the file. + // + // Two important scoping rules: + // + // 1. Reuse is disabled when any savepoint is active. A rollback_to + // would need to per-savepoint distinguish dirty entries at + // reused ids from dirty entries at preserved ids, which would + // require an 8 KB freemap snapshot per savepoint and a + // per-savepoint dirty-page list. For v1, the simpler rule is + // "reuse only outside savepoint scopes". Workloads that want + // reuse (e.g. F1 delete_subtree / drop_table) typically don't + // use savepoints at all. + // + // 2. Pages freed during the CURRENT transaction (in + // `txn_freed_pages`) are NOT reusable within the same + // transaction — their old contents must stay readable via + // `committed_roots` until commit swaps the superblock. This + // is enforced by only merging `txn_freed_pages` into + // `current_freemap` during commit, after the new roots have + // been computed. + // + // Handle-table and membership-index COW pages now share this same + // freemap-aware allocator via `cow_alloc` (each `insert`/`delete` takes an + // `alloc` closure that calls it), so they reuse freed pages before + // extending — that is what bounds their steady-state page count. Overflow + // pages still call `cache.new_page()` directly and always extend, but their + // frees feed the freemap, so a later data- or handle-table allocation can + // reclaim them. Routing overflow through the freemap too would need the + // same allocator-closure plumbing at the overflow module boundary; left as + // a v1 simplification since overflow churn is far smaller than HT churn. + /// Build a transient `FreeMapTree` handle from the current freemap roots, + /// MOVING the transaction's `freemap_session_owned` set into it so this + /// handle treats pages an earlier site already COW'd this transaction as + /// in-place-mutable. Pair with `put_freemap_tree`, which moves the (possibly + /// grown) set back out — never drop a handle from `take_` without a matching + /// `put_`, or the session set is lost and later sites re-COW. + pub(super) fn take_freemap_tree(&mut self) -> FreeMapTree { + let mut tree = FreeMapTree::from_roots( + self.current_roots.freemap_page, + self.current_roots.freemap_depth, + ); + tree.session_owned = std::mem::take(&mut self.freemap_session_owned); + tree + } + + /// Write a transient handle's grown root/depth back into the current roots, + /// move its session-owned set back into the manager, and drain its + /// COW-superseded freemap pages into `structural_superseded` (the one-commit + /// defer stream — NOT `txn_freed_pages`, since freed freemap pages are + /// recycled as structural reuse, not as data frees). + pub(super) fn put_freemap_tree(&mut self, mut tree: FreeMapTree) { + self.current_roots.freemap_page = tree.root; + self.current_roots.freemap_depth = tree.depth; + self.structural_superseded + .append(&mut tree.pending_superseded); + self.freemap_session_owned = std::mem::take(&mut tree.session_owned); + } + + pub(super) fn allocate_data_page(&mut self) -> Result { + let reuse = self.savepoints.is_empty(); + let mut tree = self.take_freemap_tree(); + let id = { + let mut cache = self.cache.borrow_mut(); + cow_alloc( + &mut cache, + &mut tree, + &mut self.freemap_hint, + &mut self.structural_reuse, + reuse, + ) + }; + // Write back tree growth + drain supersedes even on error: the freemap + // pages were extended (never freed), so on a non-fatal failure they are + // harmless above-watermark scratch, and the session set must still be + // returned so a retry/commit in the same transaction stays consistent. + self.put_freemap_tree(tree); + id + } + + /// COW `handle`'s handle-table entry to `entry`, installing the new root + /// and queuing the superseded spine pages for freemap reclamation at + /// commit. Shared by `allocate`, `update`, and `set_client_byte`. + /// + /// The superseded pages are appended to `txn_freed_pages` ONLY after the + /// new root is installed in `current_roots`: if the COW fails partway + /// (e.g. `CacheFull`), the local `freed` list is dropped and the still- + /// current old tree keeps all its pages — never freeing a live page. + pub(super) fn ht_insert(&mut self, handle: u64, entry: &HandleEntry) -> Result<()> { + let mut freed: Vec = Vec::new(); + let reuse = self.savepoints.is_empty(); + // Build the freemap-tree handle (with the session set moved in) and + // borrow the hint + structural-reuse pool as locals, all disjoint from + // `self.handle_table`, so the alloc closure (which mutates them) and the + // handle-table insert can both borrow `self` at once. + let mut tree = self.take_freemap_tree(); + let result = { + let hint = &mut self.freemap_hint; + let pool = &mut self.structural_reuse; + let mut cache = self.cache.borrow_mut(); + let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); + self.handle_table.insert( + &mut cache, + self.current_roots.handle_table_page, + handle, + entry, + &mut alloc, + &mut freed, + ) + }; + // Write back freemap growth (its supersedes go to structural_superseded + // via put_freemap_tree). Done before the `?` so a freemap COW that + // happened before an insert error still returns the session set and + // records the extended root. Handle-table supersedes (`freed`) only land + // in txn_freed_pages after the new root is installed. + self.put_freemap_tree(tree); + let new_root = result?; + self.current_roots.handle_table_page = new_root; + self.txn_freed_pages.append(&mut freed); + Ok(()) + } + + // Persist the freemap tree at commit time (ISSUES.md R2 / I11 / I18, + // generalized to the multi-page COW tree). + // + // Called once at the very start of `commit_inner`, BEFORE cache.flush(), so + // the freemap pages it COWs join the same durable write set as every other + // dirty page this transaction produced. + // + // TWO FREE-STREAMS (the load-bearing distinction the reviewer scrutinizes): + // + // * `txn_freed_pages` (DATA frees) — pages freed by this commit's + // data/handle-table/membership COW supersedes. Recorded as FREE in this + // commit's new freemap tree, so the NEXT transaction's data/HT + // allocations can reuse them. Safe to mark now: the new tree becomes + // authoritative only when this commit's superblock flips, by which point + // these pages are genuinely dead. + // + // * `structural_superseded` / `pending_structural_frees` / + // `structural_reuse` (FREEMAP-page frees) — the freemap tree's OWN COW + // supersedes. These are NOT marked free in the tree: a freemap page sits + // at a high id where the lowest-first data allocator would starve it, and + // marking a freemap page free inside the tree that is recording frees + // could cascade. Instead they ride a separate recycle: superseded this + // commit (`structural_superseded`) -> deferred one commit + // (`pending_structural_frees`, since the old page is still referenced + // until the superblock flips) -> reused as structural COW targets next + // transaction (`structural_reuse`). This makes the freemap pages ROTATE + // among a small set rather than marching the file upward ~1/commit. + // + // I18 ORDERING preserved by construction. The structural COW never draws a + // page from the freemap's own free bits (that would re-COW a leaf and + // recurse); it only ever extends the file or reuses a DEAD page from a prior + // commit (one no durable superblock still references). So a to-be-freed id + // can never be handed back to record these same frees — the I18 window + // cannot open. `persist_freemap_does_not_reuse_committed_live_pages` is the + // guardrail. + // + // DEPTH-0 EQUIVALENCE. With one leaf this reduces to: COW the leaf once + // (reusing the prior commit's dead leaf id when available, else extend), set + // the freed bits, defer the old leaf to the structural recycle. Steady-state + // page count matches the pre-tree single-page freemap. + /// Mark a single page id free in the working freemap tree, routing every + /// structural COW target through the pooled `structural_extend` (reuse a dead + /// freemap page before extending the file) and lazily materializing the + /// depth-0 root on first use. Lowers `freemap_hint` to cover `id` so the next + /// `allocate_first` scan can reach it. + /// + /// The ONE marking path shared by `persist_freemap` (this commit's data + /// frees) and `reclaim_freemap_orphans` (the defrag orphan-sweep). Both must + /// flow through the same COW + recycle discipline so the structural reuse pool + /// and supersede streams stay consistent; a second marking implementation + /// could silently diverge from the one-commit-defer crash-safety the recycle + /// depends on. Take/put the tree per call: the session-owned set and the + /// reuse pool persist on the manager across calls, so a multi-id loop still + /// COWs each leaf at most once (the session dedup carries across handles). + fn freemap_mark_free_committed_path(&mut self, id: u64) -> Result<()> { + // Take the working handle WITH the transaction's session set so a leaf an + // earlier call (or this commit's data allocations) already COW'd is + // recognized as in-place here, not re-COW'd. + let mut tree = self.take_freemap_tree(); + // RefCell so the structural-`extend` closure can drain the shared reuse + // pool by `&mut` while the rest of the method still owns `self`. + let structural_reuse = std::cell::RefCell::new(std::mem::take(&mut self.structural_reuse)); + let result = (|| { + let mut cache = self.cache.borrow_mut(); + let mut extend = + |c: &mut PageCache| structural_extend(c, &mut structural_reuse.borrow_mut()); + + // Lazy materialization: a database that has never freed a page has no + // tree yet (root == PAGE_ID_NONE). Create the depth-0 leaf now, before + // marking, since `mark_free_growing` needs a real root to COW. + // Preserve the session set across the swap. + if tree.root == PAGE_ID_NONE { + let session = std::mem::take(&mut tree.session_owned); + tree = FreeMapTree::create(&mut cache, &mut extend)?; + tree.session_owned.extend(session); + } + tree.mark_free_growing(&mut cache, id, &mut extend) + })(); + // Pull the hint back to cover `id`: the hint advances monotonically via + // `allocate_first`, so a too-high hint would start the next scan above + // `id` and never reuse it. A too-low hint only costs a wasted scan. + // (Mirrors the oracle proptest's `hint = hint.min(id)`.) + self.freemap_hint = self.freemap_hint.min(id); + // Return the (partly drained) reuse pool and write the tree back even on + // error: its COW supersedes flow to `structural_superseded` via + // put_freemap_tree; commit promotes structural_superseded + the leftover + // reuse pool into pending_structural_frees (the one-commit defer). + self.structural_reuse = structural_reuse.into_inner(); + self.put_freemap_tree(tree); + result + } + + pub(super) fn persist_freemap(&mut self) -> Result<()> { + // Nothing freed this commit => the committed tree is still exactly right, + // no COW needed. (Structural reuse / supersede streams are only ever + // non-empty when there were frees, so this single check suffices.) + if self.txn_freed_pages.is_empty() { + return Ok(()); + } + + // Mark this commit's DATA frees free in the new tree via the shared + // marking path. Each call take/puts the tree, but the session-owned set + // persists on the manager, so a leaf hit by several frees is COW'd once. + let freed: Vec = std::mem::take(&mut self.txn_freed_pages); + for id in freed.iter().copied() { + self.freemap_mark_free_committed_path(id)?; + } + self.txn_freed_pages = freed; + Ok(()) + } + + /// Reclaim freemap-typed pages orphaned by a crash that lost the in-memory + /// recycle pool. The structural recycle (decision 6 of the design) is held + /// only in memory, so a crash strands its entries: `FreeMap`/`FreeMapInterior` + /// pages that are no longer reachable from the committed tree and were never + /// marked free in the bitmap (a bounded handful — the last commit's structural + /// supersedes). This sweep walks the live tree to find the reachable set, + /// scans the file for freemap-typed pages that are neither reachable nor + /// already free, and marks each free — routing the mark through the SAME + /// `freemap_mark_free_committed_path` the commit uses (COW + recycle), so a + /// reclaimed orphan lands in the BITMAP (data-reusable), disjoint from the + /// in-memory recycle pool. Requires an active transaction (called by defrag). + /// Returns the count reclaimed. + /// + /// THE EXCLUSION SET (get this exactly right): a page in the CURRENT + /// in-memory recycle pool (`structural_reuse` ∪ `structural_superseded` ∪ + /// `pending_structural_frees`) is LIVE recycling state, NOT an orphan — + /// reclaiming it into the bitmap while it is also pool-reusable would + /// double-hand-out the page. After a crash the pool is empty, so the + /// crash-orphaned pages are correctly flagged; in a normal (no-crash) defrag + /// the live pool is excluded so the two reclamation channels never overlap. + /// + /// Reading each non-reachable page through the cache checksum-verifies it. + /// A page that fails because it is GARBAGE/corrupt (`CorruptPage` / + /// `ChecksumMismatch`) is SKIPPED, not propagated (2026-06-22 review: + /// "skip unreadable dead pages") — a non-reachable page we cannot read + /// cannot be confirmed as a freemap orphan, and a dead page's corruption is + /// irrelevant to correctness. Any OTHER read error (e.g. `IoError`, a real + /// device fault) is propagated and poisons, preserving fail-closed for true + /// hardware faults. The LIVE-tree walk (`reachable_pages`) still propagates + /// fatal on a corrupt LIVE node — only the dead-page scan is softened. The + /// scan is O(total_pages) I/O — off the hot path (defrag), bounded, and + /// acceptable. + pub(crate) fn reclaim_freemap_orphans(&mut self) -> Result { + // Skip the sweep entirely while a savepoint is active. The sweep is the + // ONLY path that COWs the freemap (draining committed-LIVE pages into the + // structural recycle streams) while a savepoint is open — ordinary + // allocation already disables structural reuse under a savepoint + // (`reuse = self.savepoints.is_empty()`). But `rollback_to` rewinds only + // the roots + cache watermark, NOT the structural streams: a page the + // sweep drained into `structural_superseded` would survive the rollback, + // get promoted at commit, and be reused as a COW target in the next + // transaction while the last-durable superblock still references it — + // silent durable freemap corruption. Deferring orphan reclamation to a + // defrag run with no active savepoint avoids the whole interaction, so + // `rollback_to_inner` correctly needs no structural-stream reset. + if !self.savepoints.is_empty() { + return Ok(0); + } + let root = self.current_roots.freemap_page; + let depth = self.current_roots.freemap_depth; + if root == PAGE_ID_NONE { + return Ok(0); // no tree yet => no freemap pages can be orphaned + } + + // Pages that are NOT orphans even though unreachable + not-free: the live + // recycle pool (all three streams). See "THE EXCLUSION SET" above. + let mut excluded: FxHashSet = FxHashSet::default(); + excluded.extend(self.structural_reuse.iter().copied()); + excluded.extend(self.structural_superseded.iter().copied()); + // Belt-and-suspenders: `begin()` clones `pending_structural_frees` + // into `structural_reuse`, so every id here is already covered by the + // `structural_reuse` term above. Kept explicitly so the exclusion + // remains correct if `begin()`'s seeding ever changes. + excluded.extend(self.pending_structural_frees.iter().copied()); + + // Collect orphan ids read-only inside a single cache-borrow scope, then + // drop the borrow before marking (the mark path re-borrows the cache). + let tree = FreeMapTree::from_roots(root, depth); + let mut orphans: Vec = Vec::new(); + { + let mut cache = self.cache.borrow_mut(); + // Upper bound: the allocation high-water (`next_page_id`), NOT the + // committed `total_pages`. After a real crash + reopen these are + // equal (open seeds next_page_id from the committed superblock), and + // every orphan — a structural supersede from a committed transaction — + // sits below it. Using next_page_id also covers a page extended + // earlier in THIS session (e.g. the forge-orphan test), which a stale + // committed total_pages would miss. + let total = cache.next_page_id(); + let reachable = tree.reachable_pages(&mut cache)?; + // Pages 0..superblock_count are superblocks; start the scan above them. + for id in self.superblock_count as u64..total { + if reachable.contains(&id) || excluded.contains(&id) { + continue; + } + // Skip a non-reachable page that is GARBAGE/corrupt rather than + // letting it poison the whole maintenance pass (2026-06-22 review + // decision: "skip unreadable dead pages"). A page that is not in + // the live tree cannot be confirmed as a freemap orphan if we + // cannot read its type, and a DEAD page's corruption does not + // affect correctness — so on `CorruptPage`/`ChecksumMismatch` we + // `continue`. We deliberately PROPAGATE every other read error + // (e.g. `IoError`): a real device fault should still surface and + // poison, not be silently swallowed. NOTE: the live-tree walk + // (`reachable_pages` above) still propagates fatal on a corrupt + // LIVE page — only the dead-page scan is softened. + let buf = match cache.get(id) { + Ok(buf) => buf, + Err(ChiselError::CorruptPage { .. } | ChiselError::ChecksumMismatch { .. }) => { + continue; + } + Err(e) => return Err(e), + }; + let ty = buf[0]; + if (ty == crate::page::PageType::FreeMap as u8 + || ty == crate::page::PageType::FreeMapInterior as u8) + && !tree.is_free(&mut cache, id)? + { + orphans.push(id); + } + } + } + // Mark each orphan free through the shared committed-marking path (COW + + // recycle), landing them in the bitmap as data-reusable space. + for id in &orphans { + self.freemap_mark_free_committed_path(*id)?; + } + Ok(orphans.len() as u64) + } +} diff --git a/src/transaction/lifecycle.rs b/src/transaction/lifecycle.rs new file mode 100644 index 0000000..03a1cfc --- /dev/null +++ b/src/transaction/lifecycle.rs @@ -0,0 +1,460 @@ +//! transaction::lifecycle — transaction state machine and poison +//! machinery: begin / commit / rollback (+ their `_inner` cores), +//! `check_alive` / `poison_on_fatal` / `is_poisoned` / +//! `force_poison_for_test` / `is_active`. Split out of `transaction.rs` +//! verbatim; see the parent module for the type and fields. + +use super::*; + +impl TransactionManager { + // --- Poison machinery (ISSUES.md I1) --- + // + // Every public entry point below follows the same wrapper pattern: + // + // pub fn foo(&mut self, ...) -> Result { + // self.check_alive()?; // fast path: refuse if already poisoned + // let result = self.foo_inner(...); + // self.poison_on_fatal(result) // poison iff the inner call returned a fatal error + // } + // + // commit() is the one exception: ANY error from the commit protocol + // poisons (not just fatal variants), because partial-commit state is + // fragile enough that we do not trust the in-memory view after a + // half-finished commit even if the variant would otherwise be + // operational. See commit() for the full reasoning. + + /// Returns Err(Poisoned) if the manager has previously seen a fatal + /// error. Called at the top of every public entry point. Cheap. + /// + /// Takes `&self` because the poison flag lives in a `Cell` + /// (F3: `read()` takes `&self`, and read paths must also check/set + /// the flag). + pub(super) fn check_alive(&self) -> Result<()> { + if self.poisoned.get() { + return Err(ChiselError::Poisoned); + } + Ok(()) + } + + /// Inspect a Result and set the poison flag if it contains a fatal + /// error. Returns the Result unchanged so the caller can `?` or return + /// it. Never fires on an Ok or on an operational error. + /// + /// Takes `&self` (not `&mut self`) because the flag is a `Cell` — + /// essential for the `&self`-taking read paths under F3. + pub(super) fn poison_on_fatal(&self, result: Result) -> Result { + if let Err(ref e) = result { + if e.is_fatal() { + self.poisoned.set(true); + } + } + result + } + + /// Force the manager into the poisoned state. Test-only hook used by + /// the I1 regression test to avoid needing a real I/O failure injection. + #[cfg(test)] + pub fn force_poison_for_test(&self) { + self.poisoned.set(true); + } + + /// True if this manager has been poisoned by a previous fatal error. + pub fn is_poisoned(&self) -> bool { + self.poisoned.get() + } + + /// Begin a new transaction. + /// + /// Single-writer: returns TransactionAlreadyActive if one is already in + /// flight. current_roots is reseeded from committed_roots so that any prior + /// (aborted) in-progress state is discarded. The dirty/freed bookkeeping is + /// cleared — this is the only place (besides commit/rollback) those vectors + /// are zeroed, so callers must not rely on them surviving a begin(). + pub fn begin(&mut self) -> Result<()> { + self.check_alive()?; + let result = self.begin_inner(); + self.poison_on_fatal(result) + } + + fn begin_inner(&mut self) -> Result<()> { + // Fail fast on read-only mounts so callers don't build up + // transaction state only to hit a ReadOnlyMode at the first + // write_page call during commit. + if self.cache.borrow().io().is_read_only() { + return Err(ChiselError::ReadOnlyMode); + } + if self.active_txn { + return Err(ChiselError::TransactionAlreadyActive); + } + self.current_roots = self.committed_roots.clone(); + // The freemap root+depth ride in current_roots (cloned just above), so + // there is no separate freemap working copy to reset here. The hint is + // untracked (a stale hint only costs a scan), so it is left as-is too. + // The session-owned set is strictly per-transaction: a page COW'd last + // transaction is now committed and must NOT be mutated in place, so start + // empty. (begin already requires no active txn, so it is normally empty, + // but clear defensively.) + self.freemap_session_owned.clear(); + // Seed the structural reuse pool from the prior commit's deferred dead + // freemap pages: those superblock-unreferenced pages are now safe to + // reuse as this transaction's freemap COW targets, so the freemap rotates + // among a bounded set instead of extending. CLONE (not move) so + // `pending_structural_frees` stays intact as the rollback fallback — a + // rolled-back transaction never reached commit, so its structural recycle + // is exactly the pre-transaction one. `commit_inner` overwrites it on the + // success path. `structural_superseded` is empty here (only + // persist_freemap fills it); clear defensively. + self.structural_reuse = self.pending_structural_frees.clone(); + self.structural_superseded.clear(); + // R1: clone the live-slot counts and reset the insert cursor. + // The cursor is always None at begin — it only tracks pages + // allocated during the current transaction. + self.current_live_slots = self.committed_live_slots.clone(); + self.insert_cursor = None; + self.active_txn = true; + self.savepoints.clear(); + self.txn_freed_pages.clear(); + Ok(()) + } + + /// Durably commit the active transaction. + /// + /// Commit protocol — ORDERING IS LOAD-BEARING. Each numbered step encodes a + /// specific crash-safety guarantee; reordering any of them can lose data or + /// expose torn state on recovery. + /// + /// A commit issues THREE fsyncs, not two: a pre-drain (step 0) plus the two + /// numbered below. Both pre-drain and step 1 are part of the "all data + /// durable BEFORE the superblock" phase — the pre-drain just moves some of + /// that flushing earlier; the superblock fsync (step 4) is the second phase. + /// + /// 0. Pre-drain the page cache (I28). BEFORE step 1, `commit_inner` flushes + /// the cache once so that `persist_freemap`'s own page allocation cannot + /// trip the spill / `CacheFull` ceiling mid-commit (which would poison on + /// an operational error). This is the FIRST of the three fsyncs. See the + /// I28 comment in `commit_inner` for why it is conditional-safe. + /// + /// 1. Flush all dirty data pages to disk AND fsync. + /// PageCache::flush() writes every dirty page then calls fsync(). After + /// this returns, every page the new superblock will reference is durable + /// on the storage medium. WHY FIRST: the new superblock is the pointer + /// that makes these pages "live". If we wrote the superblock before the + /// data pages were durable and crashed, recovery would pick up a + /// superblock whose root_handle_table_page points into a page whose + /// contents were never persisted — corruption with a valid checksum on + /// the superblock but garbage at the referenced page. + /// + /// 2. Compute the new superblock in memory. + /// Bump txn_counter first so (a) the new superblock outranks the old one + /// via Superblock::select()'s max_by_key, and (b) `txn_counter % + /// superblock_count` selects which slot to overwrite (step 3). For N=2 + /// this is the original parity alternation; for N>=3 (R4) it is true + /// round-robin across all N slots. total_pages is queried from the file + /// AFTER flush() so any new_page() allocations are reflected. + /// + /// 3. Write the new superblock to the INACTIVE slot. + /// The target is `txn_counter % superblock_count`, which always + /// points at the stalest slot. The N-1 other slots (including the + /// previously-active one, at counter txn_counter-1) are untouched + /// and still hold valid superblocks at strictly smaller counters. + /// WHY: if we crash during this write, the target slot may be torn + /// (bad checksum) but every other slot still holds the last + /// committed state (or earlier ones). Recovery picks the highest + /// surviving valid counter and the transaction is simply lost — + /// never half-applied. Overwriting an active slot in place would + /// be catastrophic: a torn write there could destroy a valid + /// superblock. Higher N buys survival of CONSECUTIVE torn writes + /// to the same target slot on retry (see `create_new` docstring). + /// + /// 4. fsync the superblock write. + /// This is the LINEARIZATION POINT of the commit. Before this fsync the + /// transaction is not durable, even if write_page returned; the kernel + /// may still be holding the superblock page in its buffer cache. After + /// this fsync returns successfully, a crash-and-recover will observe the + /// new state. A SINGLE fsync (combining data pages and superblock) would + /// be unsafe because the OS is free to reorder writes within an fsync + /// boundary — the superblock could reach the disk before the data pages + /// it references, creating a window where a crash leaves a valid-looking + /// superblock pointing at non-durable data. + /// + /// 5. Update in-memory committed_roots and clear txn state. + /// Only after the superblock fsync succeeds do we promote current_roots + /// to committed_roots. If ANY step in the protocol fails the manager is + /// poisoned (see the I1 block below) — active_txn / committed_roots are + /// left untouched but no public API will accept further calls; the only + /// legal recovery is close + reopen, which picks the last-durable + /// superblock via `Superblock::select`. Retry-in-place is forbidden + /// because a half-committed state (dirty flags already cleared in the + /// cache, txn_counter possibly bumped, target slot possibly torn on + /// disk) cannot be safely continued, and Linux fsyncgate semantics make + /// re-calling fsync() after a failed fsync unsafe regardless. + pub fn commit(&mut self) -> Result<()> { + self.check_alive()?; + // Special poison policy for commit: we refuse BOTH operational and + // fatal errors that arise after the commit protocol has started. + // The operational NoActiveTransaction case is checked BEFORE any + // protocol state is touched, so it stays operational and does not + // poison. But once cache.flush() has run, any subsequent error — + // even an otherwise operational one — leaves the manager in a + // partial-commit state (dirty flags cleared in the cache, counter + // possibly bumped, superblock possibly torn on disk) that cannot be + // safely continued. Under Linux fsyncgate semantics a failed fsync + // cannot be retried at all, so we poison and force the caller to + // reopen. + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + let result = self.commit_inner(); + if result.is_err() { + self.poisoned.set(true); + } + result + } + + fn commit_inner(&mut self) -> Result<()> { + // I27: flatten every still-active savepoint's `freed_pages` + // back into `txn_freed_pages` before persist_freemap consumes + // it. savepoint_inner moves `txn_freed_pages` INTO the + // savepoint record (via std::mem::take), so any frees that + // happened before a still-unreleased savepoint otherwise get + // dropped on the floor when step 5 calls `savepoints.clear()` + // — a permanent freemap leak for the "commit with savepoint + // active" pattern. Mirrors `release_inner`'s merge but applied + // across the full stack. We take the lists out of the + // savepoints (rather than iterating by reference) so the + // savepoints hold no stale `freed_pages` if we ever change + // step 5 to drain instead of clear; current code is equivalent + // either way. + for sp in self.savepoints.iter_mut() { + self.txn_freed_pages.append(&mut sp.freed_pages); + } + + // I28: drain the page cache BEFORE persist_freemap runs. Without + // this, `persist_freemap`'s own `allocate_data_page` can trip + // `maybe_evict`'s spill-or-CacheFull decision (every existing entry + // dirty, nothing evictable, and either spillway disabled or full) + // and return `ChiselError::CacheFull` or `ChiselError::SpillwayFull`. + // The CacheFull variant is operational-by-design (I19 docs: "caller + // recovers by + // committing or rolling back"), but commit's poison wrapper fires + // on any error once the protocol has started — demoting an + // operational signal to fatal for a caller who has no legal + // action left (commit is precisely what failed). Pre-draining + // clears every dirty pin so the ceiling is reachable via normal + // eviction when persist_freemap itself allocates. Cost: one + // extra fsync on every commit. That is consistent with the + // project's explicit "durability over performance" posture — + // the alternative reclassifies CacheFull as fatal inside commit, + // which is both more surprising and harder to document cleanly. + // + // Ordering note: this flush is safe to do before persist_freemap. + // The shadow-paging invariant requires "new-freemap-page durable + // before superblock" (step 1's flush does that). The pre-drain + // only affects user-dirty pages, which are already part of the + // transaction's durable write set — just flushed earlier. The + // subsequent step 1 flush handles the one new freemap page + // persist_freemap adds. + self.cache.borrow_mut().flush()?; + + // Step 0 (ISSUES.md R2 / I11): persist the freemap tree. This marks + // `txn_freed_pages` (plus the prior commit's deferred structural frees) + // free in a COW of the committed tree and updates + // `current_roots.{freemap_page, freemap_depth}`. Runs BEFORE the main + // flush so the new freemap pages join the same durable write set as all + // other dirty data pages. + self.persist_freemap()?; + + // Hold one RefMut for the remaining steps. Dropping and + // re-borrowing between steps would be semantically identical + // but noisier. + let mut cache = self.cache.borrow_mut(); + + // Step 1: Flush all dirty pages (PageCache::flush internally fsyncs). + // After this, every page the new superblock will reference is on disk. + cache.flush()?; + + // Step 2: Build the new superblock. Bumping txn_counter here both makes + // it outrank the current superblock on recovery AND (via parity) picks + // the target slot in step 3. + // + // I119 (ISSUES.md, 2026-06-21): checked, not `+= 1`. A wrapped counter + // would corrupt `Superblock::select`'s "highest counter wins" (release + // wrap to 0) — far worse than the loud, controlled panic here. Overflow + // needs 2^64 commits, so it is structurally unreachable; a dedicated + // fatal error variant for an impossible event would be speculative + // public surface, so the `expect` on the invariant is proportionate. + self.txn_counter = self + .txn_counter + .checked_add(1) + .expect("txn_counter overflowed u64 (2^64 commits) — unreachable"); + let total_pages = cache.file_page_count()?; + let sb = Superblock { + magic: page::MAGIC, + format_version: page::FORMAT_VERSION, + txn_counter: self.txn_counter, + root_handle_table_page: self.current_roots.handle_table_page, + root_freemap_page: self.current_roots.freemap_page, + total_pages, + next_handle: self.current_roots.next_handle, + page_size: PAGE_SIZE as u32, + named_roots: self.current_roots.named_roots, + // R4: every slot records the current N so open-time + // recovery can discover it from the winning slot without + // external hints. + superblock_count: self.superblock_count, + root_membership_index_page: self.current_roots.membership_index_page, + // Freemap tree depth, paired with root_freemap_page. 0 = today's + // single-leaf format; grows as the tree deepens. + freemap_depth: self.current_roots.freemap_depth, + }; + let buf = sb.serialize(); + // Step 3: Write to the INACTIVE slot. For N superblock slots, + // the slot is `txn_counter % N` — a round-robin that always + // targets the stalest slot. With N=2 this is the parity + // alternation from the original layout; with N>=3 it extends + // to true round-robin. The currently-active slot (and every + // other non-target slot) is never touched, so a torn write + // here can only damage the new superblock, never the N-1 + // last-known-good ones. + let inactive = self.txn_counter % self.superblock_count as u64; + cache.io_mut().write_page(inactive, &buf)?; + // Step 4: Durability linearization point. Until this fsync returns the + // transaction is not crash-safe; after it returns the new state is + // observable on recovery. + cache.io_mut().fsync()?; + + // Step 5: Promote in-memory state. Only now is the txn officially committed. + self.committed_roots = self.current_roots.clone(); + self.committed_roots.total_pages = total_pages; + // The committed freemap tree advances automatically: its {root, depth} + // ride in current_roots, promoted into committed_roots just above. No + // separate in-memory freemap copy to advance. + // R1: promote the live-slot counts. The cursor is per-transaction + // and gets reset for the next begin(). + self.committed_live_slots = self.current_live_slots.clone(); + self.insert_cursor = None; + self.active_txn = false; + self.savepoints.clear(); + // txn_freed_pages were already marked free in the new committed freemap + // tree by persist_freemap; clear the vector now that it's done its job. + self.txn_freed_pages.clear(); + // Every freemap page COW'd this transaction is now committed; the next + // transaction must COW (not edit in place) any of them it touches. + self.freemap_session_owned.clear(); + // Promote the freemap structural recycle for the next transaction: the + // pages this commit superseded (`structural_superseded`) become dead the + // instant the superblock flips above — and the reuse-pool remainder + // (`structural_reuse` ids not consumed as COW targets) is likewise still + // dead and reusable. Both become next transaction's `pending_structural_frees`. + self.pending_structural_frees.clear(); + self.pending_structural_frees + .append(&mut self.structural_superseded); + self.pending_structural_frees + .append(&mut self.structural_reuse); + + Ok(()) + } + + /// Abort the active transaction and discard all in-memory changes. + /// + /// Uses watermark-based rollback (ISSUES.md I3): `cache.truncate` is + /// called with `committed_roots.total_pages`, which both drops every + /// cache entry for pages allocated during the transaction AND truncates + /// the file back to its pre-transaction size. This fixes the earlier + /// bug where rollback would leave zeroed trailing pages in the file + /// because the cache-level discard did not propagate to `ftruncate`. + /// + /// Because `PageCache::new_page()` hands out monotonically increasing + /// ids, the pre-transaction watermark cleanly separates "pages that + /// existed at begin() time" (< watermark, preserved) from "pages + /// allocated during this transaction" (>= watermark, discarded). No + /// per-page tracking list is required. + pub fn rollback(&mut self) -> Result<()> { + self.check_alive()?; + let result = self.rollback_inner(); + self.poison_on_fatal(result) + } + + fn rollback_inner(&mut self) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + + // Rollback the cache in two steps: + // (a) Discard every dirty entry. This catches pages REUSED from + // the freemap whose id is less than the watermark — the + // watermark-based truncate below only catches extended + // pages. After discard, the next read for such a page id + // will re-load the last-committed content from disk, which + // is exactly the pre-transaction state. Safe because + // `flush()` (commit) always clears dirty flags, so any + // dirty entry was created in the current transaction. + // (b) Truncate to committed_roots.total_pages. This rewinds + // next_page_id AND shrinks the file, dropping every page + // allocated via extension (id >= watermark). Together with + // (a), this returns the cache and file to their exact + // pre-transaction state. + { + let mut cache = self.cache.borrow_mut(); + cache.discard_all_dirty(); + cache.truncate(self.committed_roots.total_pages)?; + } + + self.current_roots = self.committed_roots.clone(); + // C1: MembershipIndex.outer_depth is in-memory state that index grows + // mutate during the transaction, but it is NOT carried in Roots, so the + // snapshot restore above does not rewind it. Re-derive it from the (now + // committed) root — mirroring the open-time recovery — so the in-memory + // descent depth matches the page it descends. Otherwise handles_with_tag + // mis-descends a rolled-back-shallow root with a stale-deep depth. + { + let mut cache = self.cache.borrow_mut(); + let depth = + RadixU64::recover_depth(&mut cache, self.current_roots.membership_index_page)?; + self.membership_index.set_outer_depth(depth); + } + // I99: HandleTable.depth is the same kind of in-memory radix-depth cache + // as outer_depth above -- mutated by grows, not carried in Roots -- so it + // must also be re-derived from the restored root. Otherwise a rolled-back + // handle-table grow leaves the descent depth too deep and lookups + // mis-descend, returning InvalidHandle for committed handles. + { + let mut cache = self.cache.borrow_mut(); + let depth = + HandleTable::recover_depth(&mut cache, self.current_roots.handle_table_page)?; + self.handle_table.set_depth(depth); + } + // The freemap root+depth were restored by `current_roots = + // committed_roots.clone()` above; any dirty freemap pages this + // transaction COW'd sit above the watermark and were dropped by the + // truncate. The hint is untracked, so nothing to revert. + // + // `pending_structural_frees` is left intact: begin() CLONED it into + // `structural_reuse` rather than moving it, so it still holds the + // pre-transaction dead-freemap-page set — correct, since a rolled-back + // transaction's structural recycle is exactly the pre-transaction one. + // We DISCARD the in-transaction structural working state: + // * `structural_superseded` holds committed-tree freemap pages this + // aborted transaction COW'd-over; the abort means the committed tree + // still references them, so they are NOT dead and must never be + // recycled. + // * `structural_reuse` was the working copy; drop it. + // * the session-owned set: any freemap pages this aborted transaction + // COW'd sit above the watermark and were just truncated, so their ids + // must not be treated as in-place-mutable next transaction. + self.structural_superseded.clear(); + self.structural_reuse.clear(); + self.freemap_session_owned.clear(); + // R1: revert the live-slot counts and drop the insert cursor. + self.current_live_slots = self.committed_live_slots.clone(); + self.insert_cursor = None; + self.active_txn = false; + self.savepoints.clear(); + self.txn_freed_pages.clear(); + Ok(()) + } + + pub fn is_active(&self) -> bool { + self.active_txn + } +} diff --git a/src/transaction/mod.rs b/src/transaction/mod.rs new file mode 100644 index 0000000..06a6c3a --- /dev/null +++ b/src/transaction/mod.rs @@ -0,0 +1,312 @@ +// transaction.rs — Transaction lifecycle, savepoints, commit protocol, and data operations. +// This is the orchestration layer (layer 6 in the module graph per ARCHITECTURE.md) that ties +// together the handle table, data pages, overflow pages, freemap, superblock, and page +// cache into a coherent transactional API. +// +// Durability model (shadow paging, no WAL): +// - Writes never overwrite live pages. Mutations go to freshly allocated pages via +// PageCache::new_page() and the new roots are threaded through a rebuilt handle +// table spine (COW). The previously-committed pages remain intact on disk until +// the new superblock supersedes them. +// - A commit becomes visible atomically when a new superblock with a higher +// txn_counter and a valid checksum is fsync'd to its (alternating) slot. +// - Crash recovery = open_existing() runs Superblock::select() and picks the +// highest-txn_counter superblock with a valid checksum. A torn/partially-written +// new superblock fails its checksum, so the previous committed state wins — +// no log replay, no undo. +// +// Concurrency model: +// - A TransactionManager is single-writer. active_txn guards against nested begin(). +// Multi-process exclusion is enforced at the file layer by flock() in PageIo; +// only one TransactionManager may hold the database open at a time. +// - TransactionManager is NOT internally thread-safe — callers must serialize +// access. Readers and writers share the same PageCache; there is no MVCC. +// +// In-memory vs on-disk state during an open transaction: +// - All mutations live in the PageCache as dirty entries. Nothing mutated by the +// transaction is durable (or even written to the file in general) until commit(). +// - The superblock on disk still points at committed_roots; current_roots lives +// only in memory. A crash mid-transaction discards all dirty pages from cache +// and the on-disk superblock still references the prior committed snapshot. +// - NOTE: `new_page()` (file extension) extends the underlying file immediately; +// `allocate_data_page` prefers reuse from `current_freemap` but also calls +// through to `new_page()` when the freemap is empty. Either way, any pages +// extended-but-uncommitted before a crash are harmless because nothing in the +// committed superblock references them, and the rollback path +// (`cache.truncate(committed_roots.total_pages)` — I3) actively shrinks the +// file on a clean rollback so they don't accumulate at all. +// +// In-memory mode: `TransactionManager::create_new` and `open_existing` are +// backend-agnostic — whether the underlying PageIo is backed by a file (with +// flock) or by a Vec (no flock, no durability) is invisible here. The +// in-memory entry points live in `lib.rs` and just hand this module a +// memory-backed PageIo. Every transactional invariant in this file (commit +// ordering, poison on fatal error, watermark rollback) applies equally to the +// in-memory backend. + +use std::cell::{Cell, RefCell}; +// I127 (ISSUES.md, 2026-06-21): FxHashMap (not std SipHash) for the per-op +// slot-accounting maps below (current/committed_live_slots, Savepoint.live_slots). +// Keys are trusted local u64 page ids — no DoS surface — so SipHash is pure cost, +// exactly the I77 rationale; that pass converted the page cache/LRU but missed +// these. FxHashMap is a drop-in std HashMap with a faster non-DoS-resistant hasher. +use rustc_hash::{FxHashMap, FxHashSet}; + +use crate::data_page::DataPage; +use crate::error::{ChiselError, Result}; +use crate::freemap_tree::FreeMapTree; +use crate::handle_table::{HandleEntry, HandleFlags, HandleTable}; +use crate::membership_index::{MembershipIndex, RadixU64}; +use crate::overflow::Overflow; +use crate::page::{self, PAGE_ID_NONE, PAGE_SIZE}; +use crate::page_cache::PageCache; +use crate::stats::ChiselCounters; +use crate::superblock::{ + NamedRoot, Superblock, MAX_SUPERBLOCKS, NAMED_ROOT_COUNT, NAMED_ROOT_NAME_LEN, +}; + +// Largest value stored inline in a data-page slot. Larger values are written to an +// overflow chain and referenced by a single HandleEntry with HandleFlags::Overflow. +// +// I117 (ISSUES.md, 2026-06-21): COMPUTED from the page constants (was a +// hand-maintained `8162` literal with only a prose "keep in sync" note). A data +// page's usable body is `CHECKSUM_OFFSET - DATA_PAGE_HEADER_SIZE`, minus one +// `SLOT_ENTRY_SIZE` slot-directory entry. Deriving it makes drift impossible — +// which is what makes the `.expect("value fits in empty page")` in +// `insert_into_data_page` safe by construction: a value `<= MAX_INLINE_VALUE` +// always fits an empty page, so that expect is structurally unreachable. +const MAX_INLINE_VALUE: usize = + page::CHECKSUM_OFFSET - page::DATA_PAGE_HEADER_SIZE - crate::data_page::SLOT_ENTRY_SIZE; + +/// Snapshot of the mutable "pointers" that define a consistent database state. +/// A commit succeeds by writing a superblock that references exactly these roots; +/// a rollback succeeds by reverting current_roots back to committed_roots. +/// +/// The `named_roots` array is part of this snapshot (ISSUES.md F2) so that +/// set_root_name / clear_root_name participate in the transactional commit +/// point for free — a rollback or `rollback_to` restores named roots at +/// the same time it restores the handle-table root, with no extra plumbing. +#[derive(Debug, Clone)] +struct Roots { + handle_table_page: u64, + // Root page of the freemap tree (PageType::FreeMap leaf at depth 0, or a + // FreeMapInterior at depth > 0). PAGE_ID_NONE until the first free + // materializes the tree (see persist_freemap). Paired with `freemap_depth`, + // these two words ARE the committed freemap; cloning Roots at + // begin/commit/rollback/savepoint carries them with no extra plumbing. + freemap_page: u64, + // Depth of the freemap tree rooted at `freemap_page`. Depth 0 = today's + // single-leaf format (a lone FreeMap page reached directly), so existing + // databases load unchanged. Grows logarithmically with database size. + freemap_depth: u32, + next_handle: u64, + total_pages: u64, + named_roots: [NamedRoot; NAMED_ROOT_COUNT], + // Root page of the membership index (chunk-tags). PAGE_ID_NONE until the + // first tagged chunk is written. Cloned automatically with the rest of + // Roots at begin/commit/rollback/savepoint — no extra plumbing needed. + membership_index_page: u64, +} + +/// A nested rollback point within an active transaction. +/// +/// Captures the roots and the `next_page_id` watermark at savepoint +/// creation time. `rollback_to(name)` restores the roots and calls +/// `cache.truncate(watermark)`, which drops every cache entry and +/// truncates the file back to the watermark — cleanly discarding every +/// page the transaction allocated after the savepoint (ISSUES.md I3). +/// +/// `freed_pages` is still tracked per-savepoint so a future freemap +/// reclamation pass (R2) can restore freed-but-not-yet-reclaimed pages +/// if a savepoint is rolled back to. It is a distinct concern from the +/// cache-level rollback that the watermark handles. +/// +/// `live_slots` and `insert_cursor` snapshot the R1 packing state +/// (live slot counts per data page + the current in-progress insert +/// cursor). `rollback_to` restores these so a savepoint rewind leaves +/// the packer in a consistent state. Cloning the HashMap is O(map +/// size) per savepoint but savepoints are rare in the target workloads +/// (drop_table / delete_many don't use them). +#[derive(Debug)] +struct Savepoint { + name: String, + roots: Roots, + watermark: u64, + freed_pages: Vec, + live_slots: FxHashMap, + insert_cursor: Option, +} + +/// The single writer for a Chisel database. Not thread-safe; file-level mutual +/// exclusion across processes is provided by flock() in PageIo. Holds both the +/// last durably-committed roots (for reads outside a txn and for rollback) and +/// the in-progress current_roots (only valid while active_txn is true). +pub struct TransactionManager { + // Interior mutability (ISSUES.md F3): the page cache is mutated on read + // (LRU bookkeeping, page loads, checksum validation), but from Chisel's + // public API perspective a read() is semantically a read. Wrapping in + // RefCell lets `read()` / `handles()` / `stats()` take `&self` so + // callers don't need an external RefCell wrapper. RefCell (not + // Mutex) because Chisel is deliberately single-threaded — see + // lib.rs and ARCHITECTURE.md. Every access through this field uses + // `borrow_mut()`; reborrowing for downstream `&mut PageCache` parameters + // (e.g., handle_table methods) is done via `&mut *cache` on a single + // RefMut held for the duration of the operation. + cache: RefCell, + // Roots that match the superblock currently on disk. Safe to read at any time. + committed_roots: Roots, + // Roots under construction. Equals committed_roots when no txn is active; + // diverges from it as mutations create new COW pages during a txn. + current_roots: Roots, + handle_table: HandleTable, + /// In-memory state for the membership index (chunk tags). Holds only the + /// outer tree's depth; the root lives in current/committed `Roots`. + membership_index: MembershipIndex, + // Monotonically increasing. Written into each new superblock; the higher value + // wins on recovery. Also used to pick the inactive slot on commit via + // `txn_counter % superblock_count`. + txn_counter: u64, + // Number of superblock slots occupying pages 0..superblock_count + // (ISSUES.md R4). Set at open time from the winning superblock's + // own `superblock_count` field; cached here so commit doesn't have + // to re-fetch it. Must equal every slot's self-reported value in a + // healthy database; divergence would indicate mid-flight reconfig + // or corruption. + superblock_count: u32, + active_txn: bool, + savepoints: Vec, + // Pages whose contents are no longer reachable from the new roots. + // Merged into `current_freemap` at commit time so subsequent + // transactions can reuse the space (ISSUES.md I9 / I10 / I11 / R2). + // During the transaction itself these pages are NOT reusable — + // their old contents must stay readable via `committed_roots` until + // commit promotes the new roots. + txn_freed_pages: Vec, + // Best-effort lower bound on the lowest free page id in the committed + // freemap tree, threaded into `FreeMapTree::allocate_first` so a scan + // starts near the answer instead of at id 0. Deliberately NOT + // transactionally tracked: a too-low hint only costs a wasted left-to-right + // scan, never correctness (the scan still returns the true lowest free id), + // so it needs no begin/rollback snapshotting. `allocate_first` advances it; + // a free at a lower id is invisible to the hint until the next scan walks + // back over it, which is acceptable slack. Init 0. + freemap_hint: u64, + // Dead freemap pages carried BETWEEN commits, the engine's bounded-growth + // mechanism for the extend-only freemap (ISSUES.md I18, generalized to the + // tree). Lifecycle: + // + // * A freemap mutation (data-alloc-side leaf COW, or persist's frees) must + // COW the committed freemap pages it touches — it can never overwrite a + // page the last-durable superblock still references. Each COW supersedes + // an OLD freemap page. + // * That old page cannot be reused IN THE SAME COMMIT (the commit's new + // freemap root may still reference it until the superblock flips), so it + // is DEFERRED one commit: collected in `structural_superseded` this + // transaction, promoted to `pending_structural_frees` at commit. + // * The NEXT transaction reuses them: `begin()` moves them into + // `structural_reuse`, and every structural `extend` (freemap COW target) + // pops from that pool before extending the file. This is what makes the + // freemap leaf ROTATE among a small set of pages instead of marching the + // file upward ~1 page/commit forever. Reusing a DEAD page (vs. a free bit + // in the tree) keeps the extend-only TERMINATION guarantee — no freemap + // mutation ever draws structural space from the freemap's own bits. + // + // Not data-reusable (never enters `txn_freed_pages`): a freed freemap page + // sits at a high id, and the lowest-first data allocator would starve it, so + // routing it back as structural reuse (where demand matches supply at steady + // state) is what actually reclaims it. + pending_structural_frees: Vec, + // The dead-freemap-page pool available to reuse as structural COW targets in + // the CURRENT transaction. Seeded from `pending_structural_frees` at + // `begin()`; drained by every structural `extend`; the unconsumed remainder + // is carried forward (back into `pending_structural_frees`) at commit. On + // rollback it is moved back wholesale, restoring the pre-transaction + // `pending_structural_frees`. + structural_reuse: Vec, + // This transaction's freemap-COW supersedes (old freemap pages this txn + // replaced). Accumulated as transient handles drain `tree.pending_superseded` + // here via `put_freemap_tree`; promoted to `pending_structural_frees` at + // commit (the one-commit defer). Dropped on rollback (those COWs are + // truncated above the watermark). + structural_superseded: Vec, + // Freemap pages already COW'd/extended by the CURRENT transaction. Because + // the manager rebuilds a transient `FreeMapTree` handle at every allocation + // site (data-page alloc, each HT/membership COW, persist_freemap), this set + // is what lets those handles share the "first touch this txn => COW, later + // touches => in-place" discipline: without it every site would re-COW the + // same freemap leaf, turning reclamation into unbounded file growth. Swapped + // into each transient handle and read back out (see `freemap_tree` helper). + // Cleared at begin (fresh per transaction); also cleared on commit/rollback + // so the next transaction starts empty. A stale entry pointing at a + // now-committed page would be a CORRECTNESS bug (it would suppress a needed + // COW and mutate a live committed page in place), which is exactly why it is + // transaction-scoped, not cross-transaction. + freemap_session_owned: FxHashSet, + // Live-slot count per data page (ISSUES.md R1). Tracks how many + // handle-table entries currently point at each data page — this + // is the information needed to decide when a page is fully empty + // and can be returned to the freemap. `committed_live_slots` is + // the durable state (rebuilt at open time by scanning the handle + // table); `current_live_slots` is the in-transaction working copy. + // + // Kept in memory rather than on disk because updating a slot count + // on a committed data page would require COW, and COWing a data + // page would require rewriting every handle_table entry that + // points into it — an O(live-slots-in-page) amplification per + // delete that shadow paging does not handle well. + committed_live_slots: FxHashMap, + current_live_slots: FxHashMap, + // Per-transaction "insert cursor" (ISSUES.md R1). The id of a data + // page allocated earlier in the current transaction that still has + // free space. New values pack into it until it fills, at which + // point a new page is allocated and becomes the new cursor. + // + // `None` at the start of each transaction. Only set for pages + // allocated during THIS transaction (so they're dirty in the cache + // and safe to modify). A committed data page is never the cursor — + // that would require COW, which is prohibitively expensive for data + // pages (every handle_table entry pointing at the page would need + // to be rewritten). Disabled entirely when savepoints are active, + // same as freemap reuse (R2): the savepoint-snapshot cost becomes + // manageable when only one code path interacts with packing state. + insert_cursor: Option, + // Poison flag (ISSUES.md I1). Once set, every public entry point returns + // ChiselError::Poisoned until the manager is dropped. Set by commit() on + // any error in the commit protocol, and by `poison_on_fatal()` for any + // fatal error observed during other operations. Modeled on + // std::sync::Mutex poisoning: the only legal recovery is to drop the + // Chisel handle and reopen; the shadow-paging crash-recovery logic then + // returns the database to the last durable state. Linux fsync semantics + // (fsyncgate, 2018) make this the ONLY safe response to a mid-commit + // I/O error — a failed fsync cannot be retried without first closing + // and reopening the file. + // + // Stored as `Cell` (not plain `bool`) so it can be set from the + // `&self`-taking read paths introduced by F3. Cell rather than + // AtomicBool because TransactionManager is !Sync by design (see + // lib.rs); there is no cross-thread access to synchronize against. + poisoned: Cell, + + // Test-only fault injection consolidated off the production type (review + // 2026-06-22 SMELL #4): the four BUG#2 atomic-staging arming flags live in + // their own `#[cfg(test)]` struct so they carry no production scaffolding + // fields here. See `fault.rs` for each Cell's precise divergence window. + #[cfg(test)] + fault: fault::FaultInjector, +} + +mod config; +#[cfg(test)] +mod fault; +mod freemap; +mod lifecycle; +mod mutate; +mod named_roots; +mod packing; +mod read; +mod recovery; +mod savepoints; +mod staging; +mod stats; +#[cfg(test)] +mod tests; diff --git a/src/transaction/mutate.rs b/src/transaction/mutate.rs new file mode 100644 index 0000000..13f4719 --- /dev/null +++ b/src/transaction/mutate.rs @@ -0,0 +1,415 @@ +//! transaction::mutate — mutation API: update / delete / delete_tagged / +//! delete_with_tag / delete_many (+ their `_inner` cores). Split out of +//! `transaction.rs` verbatim; see the parent module for the type and fields. + +use super::freemap::cow_alloc; +use super::*; + +impl TransactionManager { + /// Update an existing handle to point at a new value. + /// + /// Allocates a new slot/overflow chain for the new value and rewrites + /// the HandleEntry via COW. The OLD location is retired differently + /// depending on its kind: + /// + /// * Inline (Live): goes through `release_data_slot`, which + /// decrements the per-page live-slot count (R1). Only when the + /// count reaches zero does the entire page land in + /// `txn_freed_pages`. Otherwise the slot becomes a tombstone, + /// reclaimable only via defrag (R3). + /// * Overflow: the whole chain is deleted and every page in the + /// chain is pushed onto `txn_freed_pages`. + /// + /// The earlier "assumes one live slot per page / must change when R1 + /// lands" caveat is OBSOLETE — R1 has landed and the slot-level + /// accounting below implements exactly the post-R1 contract. + pub fn update(&mut self, handle: u64, value: &[u8]) -> Result<()> { + self.check_alive()?; + let result = self.update_inner(handle, value); + self.poison_on_fatal(result) + } + + fn update_inner(&mut self, handle: u64, value: &[u8]) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + + // `update` requires an active txn (checked above), so `lookup_live`'s + // read-view root is `current_roots` — read-your-own-writes. + let entry = self.lookup_live(handle)?; + + // Atomic staging (same discipline as delete_inner): do NOT retire the + // OLD value's storage until the NEW entry is durably installed. The + // previous "free old first" ordering meant a non-fatal CacheFull during + // the new-value write or the handle-table install left the committed + // handle still pointing at pages already queued for reclamation — a + // reachable-but-free page that commit then frees, corrupting the live + // value on the next freemap reuse. We compute the new value, the new + // handle-table root, and the old-location free set in a fallible PREPARE + // phase that touches no installed state, then install the new entry and + // retire the old location together in an infallible INSTALL phase. A + // mid-prepare failure is a complete no-op. + // + // `update` replaces an EXISTING handle, so handle_table.insert never + // grows (handle < capacity) — no in-memory depth save is needed (unlike + // allocate_inner). + + // Test-only injection (see `fail_next_update_value_write`): now the FIRST + // fallible step, so a simulated failure here retires nothing. + #[cfg(test)] + if self.fault.fail_next_update_value_write.replace(false) { + return Err(ChiselError::CacheFull { limit: 0 }); + } + + // PREPARE: write the new value storage. Tags and the client byte are + // entry-resident and carried forward unchanged (the handle is unchanged, + // so the membership index needs no edit — only the value's storage + // moves). Capture the inline data page so a later prepare failure can + // release it, keeping live-slot / cursor bookkeeping consistent. + let mut new_inline_page: Option = None; + let new_entry = if value.len() > MAX_INLINE_VALUE { + let first_page = { + let mut cache = self.cache.borrow_mut(); + Overflow::write(&mut cache, value)? + }; + HandleEntry { + page_id: first_page, + slot_index: 0, + flags: HandleFlags::Overflow, + tag: entry.tag, + client_byte: entry.client_byte, + } + } else { + let (data_page_id, slot) = self.insert_into_data_page(value)?; + new_inline_page = Some(data_page_id); + HandleEntry { + page_id: data_page_id, + slot_index: slot, + flags: HandleFlags::Live, + tag: entry.tag, + client_byte: entry.client_byte, + } + }; + + // PREPARE: compute the new handle-table root (no install). On failure, + // release the just-reserved inline slot so live-slot accounting stays + // consistent with the un-installed root (overflow new-value pages are the + // bounded commit-after-error leak class); the OLD location is untouched. + let mut ht_freed: Vec = Vec::new(); + let ht_new_root = + match self.handle_table_insert_candidate(handle, &new_entry, &mut ht_freed) { + Ok(r) => r, + Err(e) => { + if let Some(page_id) = new_inline_page { + self.release_data_slot(page_id); + } + return Err(e); + } + }; + + // PREPARE: compute the OLD location's free set without applying it. For + // Overflow this is a read-only walk of the old chain (a fallible + // cold-page load); for Live the page id is released in the install phase; + // Deleted carries no storage. On the overflow-walk failure, unwind the + // new inline slot — the old chain is untouched (the walk frees nothing). + enum OldRelease { + Inline(u64), + Overflow(Vec), + Nothing, + } + let old_release = match entry.flags { + HandleFlags::Live => OldRelease::Inline(entry.page_id), + HandleFlags::Overflow => { + let walked = { + let mut cache = self.cache.borrow_mut(); + Overflow::collect_chain_pages(&mut cache, entry.page_id) + }; + match walked { + Ok(freed) => OldRelease::Overflow(freed), + Err(e) => { + if let Some(page_id) = new_inline_page { + self.release_data_slot(page_id); + } + return Err(e); + } + } + } + HandleFlags::Deleted => OldRelease::Nothing, + }; + + // INSTALL phase (infallible): install the NEW entry first so the handle + // points at the new storage, THEN retire the OLD location (now genuinely + // unreferenced). For Live, release_data_slot does R1 slot accounting + // (freeing the page only when its last live slot goes); for Overflow, the + // walked chain pages are queued for reclamation. + self.current_roots.handle_table_page = ht_new_root; + self.txn_freed_pages.append(&mut ht_freed); + match old_release { + OldRelease::Inline(page_id) => self.release_data_slot(page_id), + OldRelease::Overflow(freed) => self.txn_freed_pages.extend_from_slice(&freed), + OldRelease::Nothing => {} + } + + Ok(()) + } + + /// Delete a handle. + /// + /// Retires the old location (inline slot via `release_data_slot` + /// with its R1 slot-level accounting; overflow chain by deleting + /// every page in the chain into `txn_freed_pages`) and then asks + /// the handle table to remove the mapping via COW. The earlier + /// "whole-page free assumes one value per page" caveat referenced + /// by `update`'s docstring is obsolete post-R1. + pub fn delete(&mut self, handle: u64) -> Result<()> { + self.check_alive()?; + let result = self.delete_inner(handle); + self.poison_on_fatal(result) + } + + fn delete_inner(&mut self, handle: u64) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + + // BUG#2 atomic staging (see allocate_inner): the FORWARD map (the + // handle-table tombstone) and the REVERSE map (membership-index removal) + // must become durable together. We compute both candidate roots — plus + // the fallible part of value-storage release — in a PREPARE phase that + // never touches `current_roots`, then install everything in an + // infallible phase. A non-fatal CacheFull/SpillwayFull mid-prepare + // leaves the delete a complete no-op, so the reverse index can never + // retain a member for a tombstoned handle — the stale entry that would + // otherwise later escalate to a fatal CorruptPage. + // + // No in-memory depth save is needed here (unlike allocate): handle-table + // DELETE never grows the tree, and MembershipIndex::remove writes its + // outer_depth back only on the success path. + + // FORWARD map: compute the tombstoned root; do NOT install yet. A single + // tree walk (I32) COWs the leaf with a tombstone and returns the + // previous entry. Returns (root, None) — unchanged root, no COW — if the + // handle was absent or already a tombstone; we escalate None to + // InvalidHandle to preserve the public-API behavior. + let mut ht_freed: Vec = Vec::new(); + let reuse = self.savepoints.is_empty(); + let mut tree = self.take_freemap_tree(); + let delete_result = { + let hint = &mut self.freemap_hint; + let pool = &mut self.structural_reuse; + let mut cache = self.cache.borrow_mut(); + let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); + self.handle_table.delete( + &mut cache, + self.current_roots.handle_table_page, + handle, + &mut alloc, + &mut ht_freed, + ) + }; + // Install freemap growth (supersedes go to structural_superseded). Done + // BEFORE the `?` so a delete that COW'd the freemap leaf yet then errored + // still records the extended root and returns the session set. + self.put_freemap_tree(tree); + let (ht_new_root, prev_entry) = delete_result?; + let entry = prev_entry.ok_or(ChiselError::InvalidHandle(handle))?; + + // Stage the value-storage release. The only FALLIBLE part — walking an + // overflow chain to collect its page ids — runs here in prepare; + // discarding the result on a later failure is safe (the still-current + // old entry keeps referencing the chain). The actual free-queueing / + // slot release is deferred to the install phase below. Order vs. the + // tombstone write does not matter for correctness — both become durable + // (or roll back) atomically at commit. + enum PendingRelease { + Inline(u64), + Overflow(Vec), + } + let release = match entry.flags { + HandleFlags::Live => PendingRelease::Inline(entry.page_id), + HandleFlags::Overflow => { + let freed = { + let mut cache = self.cache.borrow_mut(); + Overflow::collect_chain_pages(&mut cache, entry.page_id)? + }; + PendingRelease::Overflow(freed) + } + HandleFlags::Deleted => { + // I45 (ISSUES.md, 2026-05-22): a Deleted entry that `ok_or` + // didn't catch means the in-memory state contradicts itself — + // handle_table::delete returns None for already-tombstoned + // handles and the ok_or above converts None into the typed + // error, so reaching this arm signals a broken cross-module + // contract (most likely a future refactor of delete). Surface + // it typed rather than aborting the caller's process. Returned + // BEFORE any install, so current_roots stays untouched. + return Err(ChiselError::CorruptPage { + page_id: entry.page_id, + }); + } + }; + + // REVERSE map: compute the membership-removed root; do NOT install yet. + // The tag comes from the tombstoned entry. Tag 0 (untagged) is never in + // the index, so there is nothing to remove. + let mut mi_new_root: Option = None; + let mut idx_freed: Vec = Vec::new(); + if entry.tag != 0 { + // Test-only injection (see `inject_membership_failure`): simulate a + // non-fatal CacheFull at the reverse-map step so regression tests + // exercise the REAL failure handling below. No production artifact: + // the non-test `let res` is the only one compiled outside tests. + #[cfg(test)] + let res: Result<(u64, bool)> = if self.inject_membership_failure() { + Err(ChiselError::CacheFull { limit: 0 }) + } else { + self.membership_remove_candidate(entry.tag, handle, &mut idx_freed) + }; + #[cfg(not(test))] + let res: Result<(u64, bool)> = + self.membership_remove_candidate(entry.tag, handle, &mut idx_freed); + + let (new_index_root, removed) = res?; + // A tagged live handle always carries a reverse-index entry: + // allocate_tagged installs it atomically (BUG#2 / PR #40) and tags + // are immutable. So its removal must report present; a false here + // means the forward and reverse maps diverged from some OTHER + // source. Debug-only on purpose: a file committed while BUG#2 was + // still live (pre-#40) could carry a real on-disk divergence, and + // the open path gates MAJOR version only — such a legacy file must + // stay openable and a delete of its diverged handle must remain + // recoverable, so this never gates release builds. + debug_assert!( + removed, + "membership index diverged: tagged handle {handle} (tag {}) had no reverse entry", + entry.tag + ); + mi_new_root = Some(new_index_root); + } + + // INSTALL phase (infallible): tombstone + value release + reverse-map + // removal all become visible together. The superseded handle-table + // spine pages are queued only now, post-install, so an early prepare + // failure leaves them referenced by the still-current old tree. + self.current_roots.handle_table_page = ht_new_root; + self.txn_freed_pages.append(&mut ht_freed); + match release { + PendingRelease::Inline(page_id) => self.release_data_slot(page_id), + PendingRelease::Overflow(freed) => self.txn_freed_pages.extend_from_slice(&freed), + } + if let Some(root) = mi_new_root { + self.current_roots.membership_index_page = root; + self.txn_freed_pages.append(&mut idx_freed); + } + + Ok(()) + } + + pub fn delete_tagged(&mut self, handle: u64, tag: u32) -> Result<()> { + self.check_alive()?; + let result = self.delete_tagged_inner(handle, tag); + self.poison_on_fatal(result) + } + + fn delete_tagged_inner(&mut self, handle: u64, tag: u32) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + // Lookup-then-delete: verify the tag before mutating anything, so a wrong + // tag leaves both the chunk and the membership index untouched. The extra + // lookup walk (delete_inner walks again) is the price of verify-before-mutate. + // `lookup_live` rejects an absent/tombstoned handle with `InvalidHandle` + // (I125) BEFORE the tag comparison — a dead handle never surfaces as + // TagMismatch. + let actual = self.lookup_live(handle)?.tag; + if actual != tag { + return Err(ChiselError::TagMismatch { + handle, + expected: tag, + actual, + }); + } + self.delete_inner(handle) + } + + /// Bounded relation drop. See `Chisel::delete_with_tag` for the full + /// contract. Error semantics: a mid-pass `delete_inner` failure propagates + /// `Err` and the partial `TagDropProgress` is dropped — the deleted-this- + /// pass set is not reported. Each `delete_inner` is atomic (BUG#2 staging), + /// so the surviving in-transaction state is consistent (rollback or commit + /// are both safe); only the progress *reporting* is lost on error. + pub fn delete_with_tag(&mut self, tag: u32, max: usize) -> Result<(Vec, bool)> { + self.check_alive()?; + let result = self.delete_with_tag_inner(tag, max); + self.poison_on_fatal(result) + } + + fn delete_with_tag_inner(&mut self, tag: u32, max: usize) -> Result<(Vec, bool)> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + if max == 0 { + return Ok((Vec::new(), false)); + } + // Bounded enumeration: ask for max+1 so the count tells us whether more + // remain (len > max => not complete). saturating_add keeps the absurd + // max == usize::MAX case meaning "enumerate everything" instead of + // wrapping to 0 — which would enumerate nothing and falsely report the + // tag complete. The members snapshot is taken BEFORE the deletions, then + // each is deleted via delete_inner (which removes it from the index and + // frees its chunk). + // + // The snapshot must be MATERIALIZED into an owned Vec, not a live + // iterator: each delete_inner COWs the membership-index root (via + // membership_remove_candidate), so walking the index while deleting from + // it would descend a tree being rewritten underneath the walk. Collecting + // first decouples enumeration from the mutation it drives. + let members = { + let root = self.current_roots.membership_index_page; + let mut cache = self.cache.borrow_mut(); + self.membership_index.handles_for_tag_bounded( + &mut cache, + root, + tag, + max.saturating_add(1), + )? + }; + let complete = members.len() <= max; + let take: Vec = members.into_iter().take(max).collect(); + for &h in &take { + self.delete_inner(h)?; + } + Ok((take, complete)) + } + + /// Delete many handles in a single transaction. + /// + /// Today: this is a loop over `delete_inner`. After PR-A's fusion + /// (I32), each delete walks the handle table once per handle. For + /// dense delete patterns (many handles in the same leaf), a + /// per-leaf batched implementation would walk once per leaf + /// instead — that's tracked as I33 in ISSUES.md, deferred until + /// a workload demonstrates the win is worth the complexity. + /// + /// Error semantics: on the first error the loop stops and returns + /// the error. Handles deleted before the failure remain marked + /// for deletion in `current_roots`, so the caller can choose + /// between `rollback()` (abandon the whole batch) or `commit()` + /// (keep the partial work). + pub fn delete_many(&mut self, handles: &[u64]) -> Result<()> { + self.check_alive()?; + let result = self.delete_many_inner(handles); + self.poison_on_fatal(result) + } + + fn delete_many_inner(&mut self, handles: &[u64]) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + // See I33 in ISSUES.md for the deferred per-leaf batching work. + for &handle in handles { + self.delete_inner(handle)?; + } + Ok(()) + } +} diff --git a/src/transaction/named_roots.rs b/src/transaction/named_roots.rs new file mode 100644 index 0000000..1f97e77 --- /dev/null +++ b/src/transaction/named_roots.rs @@ -0,0 +1,123 @@ +//! transaction::named_roots — named-root table (ISSUES.md F2): +//! name encoding/validation plus set / get / clear (+ their `_inner` +//! cores). Split out of `transaction.rs` verbatim; see the parent module. + +use super::*; + +impl TransactionManager { + // --- Named roots (ISSUES.md F2) --- + // + // The named-root table lives inside the superblock (see + // `superblock::NamedRoot`). Modifications update + // `current_roots.named_roots` in memory; on commit that array is + // copied into the new Superblock and fsync'd along with the rest. + // On rollback or `rollback_to`, the usual snapshot restore reverts + // named roots alongside the handle-table root — no extra plumbing. + // + // Name validation is intentionally strict: names must be non-empty, + // must fit in NAMED_ROOT_NAME_LEN bytes, must not contain NUL + // (because NUL is the "empty slot" sentinel), and must be valid + // UTF-8 at the API boundary. Names are compared byte-for-byte after + // validation; the fixed 24-byte buffer is NUL-padded. + + /// Validate a root name and return its byte form, padded to + /// NAMED_ROOT_NAME_LEN with trailing NULs. Returns `InvalidRootName` + /// on any violation. + fn encode_root_name(name: &str) -> Result<[u8; NAMED_ROOT_NAME_LEN]> { + let bytes = name.as_bytes(); + if bytes.is_empty() || bytes.len() > NAMED_ROOT_NAME_LEN { + return Err(ChiselError::InvalidRootName); + } + if bytes.contains(&0) { + return Err(ChiselError::InvalidRootName); + } + let mut encoded = [0u8; NAMED_ROOT_NAME_LEN]; + encoded[..bytes.len()].copy_from_slice(bytes); + Ok(encoded) + } + + /// Bind `name` to `handle` in the named-root table. If `name` already + /// exists, its handle is overwritten. If it doesn't exist and the + /// table has no empty slots, returns `RootNameTableFull`. Requires an + /// active transaction and becomes durable on commit; reverts on + /// rollback/rollback_to. + pub fn set_root_name(&mut self, name: &str, handle: u64) -> Result<()> { + self.check_alive()?; + let result = self.set_root_name_inner(name, handle); + self.poison_on_fatal(result) + } + + fn set_root_name_inner(&mut self, name: &str, handle: u64) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + let encoded = Self::encode_root_name(name)?; + + // First pass: update in place if the name already exists. + for entry in self.current_roots.named_roots.iter_mut() { + if !entry.is_empty() && entry.name == encoded { + entry.handle = handle; + return Ok(()); + } + } + // Second pass: install in the first empty slot. + for entry in self.current_roots.named_roots.iter_mut() { + if entry.is_empty() { + entry.name = encoded; + entry.handle = handle; + return Ok(()); + } + } + Err(ChiselError::RootNameTableFull) + } + + /// Look up a named root. Returns `Ok(None)` if the name is not bound. + /// Reads see the transactional view: inside an active transaction, + /// pending `set_root_name` / `clear_root_name` changes are visible; + /// outside a transaction, reads the last durably committed table. + /// + /// Takes `&self` — named-root reads are semantically read-only. + pub fn get_root_name(&self, name: &str) -> Result> { + self.check_alive()?; + let result = self.get_root_name_inner(name); + self.poison_on_fatal(result) + } + + fn get_root_name_inner(&self, name: &str) -> Result> { + let encoded = Self::encode_root_name(name)?; + let table = if self.active_txn { + &self.current_roots.named_roots + } else { + &self.committed_roots.named_roots + }; + for entry in table.iter() { + if !entry.is_empty() && entry.name == encoded { + return Ok(Some(entry.handle)); + } + } + Ok(None) + } + + /// Remove a named root. No-op if the name is not bound (returns Ok). + /// Requires an active transaction. Becomes durable on commit; + /// reverts on rollback/rollback_to. + pub fn clear_root_name(&mut self, name: &str) -> Result<()> { + self.check_alive()?; + let result = self.clear_root_name_inner(name); + self.poison_on_fatal(result) + } + + fn clear_root_name_inner(&mut self, name: &str) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + let encoded = Self::encode_root_name(name)?; + for entry in self.current_roots.named_roots.iter_mut() { + if !entry.is_empty() && entry.name == encoded { + *entry = NamedRoot::EMPTY; + return Ok(()); + } + } + Ok(()) + } +} diff --git a/src/transaction/packing.rs b/src/transaction/packing.rs new file mode 100644 index 0000000..56dec7d --- /dev/null +++ b/src/transaction/packing.rs @@ -0,0 +1,161 @@ +//! transaction::packing — R1 data-page slot packing: releasing a data +//! slot, lazily materializing the handle table, and inserting a value into +//! a data page. Split out of `transaction.rs` verbatim; see the parent +//! module for the type and fields. + +use super::*; + +impl TransactionManager { + // --- Private helpers --- + + /// Release one slot from a data page (ISSUES.md R1). Decrements + /// `current_live_slots[page_id]`; if the count reaches zero, the + /// whole page becomes unreferenced and is pushed to + /// `txn_freed_pages` so commit can return it to the freemap. + /// Otherwise the slot becomes a tombstone: dead weight inside a + /// still-live page, reclaimable only via defrag. + /// + /// If the page is somehow not tracked in `current_live_slots` (a + /// bug; open-time scan should catch every live data page), this is + /// a no-op — we prefer leaking to a spurious free. + /// + /// NOTE: a stray orphaned line "Lazily create a handle table root + /// on first insert. A fresh database has" previously sat at the + /// top of this doc block (an interleaved remnant of + /// `ensure_handle_table`'s docstring); removed 2026-04-17 during + /// the commenting pass. The counterpart ("root_handle_table_page + /// == PAGE_ID_NONE; we don't materialize...") still sits above + /// `ensure_handle_table` below — both belong together. + pub(super) fn release_data_slot(&mut self, page_id: u64) { + let Some(count) = self.current_live_slots.get_mut(&page_id) else { + return; + }; + if *count > 0 { + *count -= 1; + } + if *count == 0 { + self.current_live_slots.remove(&page_id); + // If this page is the active insert cursor, clear the + // cursor — it's about to become free space, and we don't + // want future inserts to pack into it and then find it + // disappearing at commit time. + if self.insert_cursor == Some(page_id) { + self.insert_cursor = None; + } + self.txn_freed_pages.push(page_id); + } + } + + /// Lazily create a handle table root on first insert. A fresh + /// database has `root_handle_table_page == PAGE_ID_NONE`; we don't + /// materialize the root until there is a handle to put in it, so + /// empty databases never pay for a handle-table page. No per-page + /// rollback bookkeeping — the watermark rollback mechanism (I3) + /// handles any page allocated here automatically. + pub(super) fn ensure_handle_table(&mut self) -> Result<()> { + if self.current_roots.handle_table_page == PAGE_ID_NONE { + let root = { + let mut cache = self.cache.borrow_mut(); + self.handle_table.create_root(&mut cache)? + }; + self.current_roots.handle_table_page = root; + } + Ok(()) + } + + /// Place a value in a data page and return (page_id, slot_index). + /// + /// Post-R1 packing model: the transaction maintains an "insert + /// cursor" — a data page allocated earlier in THIS transaction + /// that still has space — and packs successive small-value inserts + /// into it until it fills. When the cursor is absent/full, a new + /// page is allocated (via `allocate_data_page`, which prefers + /// freemap reuse over file extension — R2) and becomes the new + /// cursor. Packing is disabled while savepoints are active: the + /// cursor is force-cleared by `savepoint()` and is NOT set when a + /// new page is allocated inside a savepoint scope, so each insert + /// under a savepoint gets its own page (the pre-R1 behavior). This + /// keeps the per-savepoint snapshot cheap to restore. + /// + /// Checksum is stamped eagerly after every mutation so the page carries a + /// valid internal checksum before any path could write it to the main + /// file — either the `flush` `write_page` at commit, or a spill-then-drain + /// write (an LRU-pressured dirty page is spilled to the spillway and later + /// drained back out to the main file). The next cold-load + /// (`page_cache::load_page`) verifies that checksum. + /// + /// Note: the spillway *transfer* does NOT rely on this. `rehydrate` + /// verifies the spillway's own per-slot checksum (`spillway::slot_checksum`), + /// never the page's internal bytes 8184..8192 — so a spilled page round-trips + /// safely whether or not its internal checksum is current. The internal + /// checksum only matters on the way to the main file. + /// + /// I78 proposes deferring this re-stamp to flush/drain time so a packed page + /// is hashed once, not once per value (a large bulk-insert win on fast + /// storage). It is deferred pending a benchmark; the difficulty is exactly + /// the spill-then-drain path, which would then have to re-stamp before the + /// main-file write. See ISSUES.md. + /// + /// Live-slot bookkeeping: every successful insert increments + /// `current_live_slots[page_id]`. `delete`/`update` consult this + /// map (via `release_data_slot`) to decide when a page is fully + /// empty and can be freed back to the freemap on commit. The map + /// is kept purely in memory — storing a slot count ON the data + /// page would force a COW (and a handle-table rewrite for every + /// entry pointing into it) on every delete. + pub(super) fn insert_into_data_page(&mut self, value: &[u8]) -> Result<(u64, u16)> { + // Packing path: try to reuse the current cursor page if it + // has room. The cursor only exists when savepoints are empty + // (see savepoint_inner) so this branch implicitly respects + // the "no packing under savepoints" rule. + if let Some(cursor_page_id) = self.insert_cursor { + let slot_option = { + let mut cache = self.cache.borrow_mut(); + let buf = cache.get_mut(cursor_page_id)?; + let result = DataPage::insert(buf, value); + if result.is_some() { + page::stamp_checksum(buf); + } + result + }; + if let Some(slot) = slot_option { + *self.current_live_slots.entry(cursor_page_id).or_insert(0) += 1; + return Ok((cursor_page_id, slot)); + } + // Cursor page is full. Fall through to allocate a new one; + // the new page becomes the new cursor. + } + + // Allocate a fresh data page. Under active savepoints, the + // cursor stays None (set below, then cleared by the savepoint + // check in subsequent calls) so each insert gets its own page — + // matching the pre-R1 "one value per page" behavior within + // savepoint scopes, which is the price of keeping rollback_to + // semantics simple. + let page_id = self.allocate_data_page()?; + let slot = { + let mut cache = self.cache.borrow_mut(); + let buf = cache.get_mut(page_id)?; + DataPage::init_page(buf); + // I46 INVARIANT: DataPage::insert can only return None for + // "no room"; the page was just init'd via DataPage::init_page + // (empty), and the value's length was already checked against + // MAX_INLINE_VALUE upstream (the overflow path catches anything + // larger before we get here). If DataPage::insert ever grows + // other failure modes, this expect needs to translate them to + // typed errors instead of panicking. + let slot = DataPage::insert(buf, value).expect("value fits in empty page"); + page::stamp_checksum(buf); + slot + }; + + // Only install the new page as the cursor if we're outside any + // savepoint scope. During a savepoint scope the cursor stays + // None so packing is effectively disabled. + if self.savepoints.is_empty() { + self.insert_cursor = Some(page_id); + } + *self.current_live_slots.entry(page_id).or_insert(0) += 1; + Ok((page_id, slot)) + } +} diff --git a/src/transaction/read.rs b/src/transaction/read.rs new file mode 100644 index 0000000..b28bb61 --- /dev/null +++ b/src/transaction/read.rs @@ -0,0 +1,175 @@ +//! transaction::read — read-side API: read / tag / client_byte / +//! handles / handles_with_tag (+ their `_inner` cores) and the live-handle +//! lookup helpers. Split out of `transaction.rs` verbatim; see the parent +//! module for the type and fields. + +use super::*; + +impl TransactionManager { + pub fn tag(&self, handle: u64) -> Result { + self.check_alive()?; + let result = self.tag_inner(handle); + self.poison_on_fatal(result) + } + + /// The handle-table root for the *current read view*: the in-progress + /// `current_roots` while a transaction is active (read-your-own-writes), + /// otherwise the last durably-committed `committed_roots`. Returns + /// `PAGE_ID_NONE` for an empty database — read paths guard on that + /// before walking the tree. Centralizes the snapshot selection shared + /// by every read-path helper (`tag`, `client_byte`, `read`, `handles`, + /// `handle_live_page_id`). + pub(super) fn live_handle_table_root(&self) -> u64 { + if self.active_txn { + self.current_roots.handle_table_page + } else { + self.committed_roots.handle_table_page + } + } + + /// Look up a handle that must be live, applying the "deleted ⇒ + /// `InvalidHandle`" rule in ONE place (I125). Every read/mutation entry + /// point that needs a live `HandleEntry` — `read`, `tag`, `client_byte`, + /// `set_client_byte`, `update`, `delete_tagged` — goes through here, so the + /// liveness invariant cannot drift between callers. + /// + /// `handle_table::lookup` already collapses a tombstone (and an empty/absent + /// tree, via `live_handle_table_root` returning `PAGE_ID_NONE`) to `None`, + /// so the `ok_or` below is the single site that raises the operational + /// `InvalidHandle`. Callers that want "absent is not an error" (e.g. + /// `handle_live_page_id`, which returns `Ok(None)`) deliberately do NOT use + /// this and keep their own Option-returning lookup. + pub(super) fn lookup_live(&self, handle: u64) -> Result { + let root = self.live_handle_table_root(); + let mut cache = self.cache.borrow_mut(); + self.handle_table + .lookup(&mut cache, root, handle)? + .ok_or(ChiselError::InvalidHandle(handle)) + } + + fn tag_inner(&self, handle: u64) -> Result { + Ok(self.lookup_live(handle)?.tag) + } + + /// Return the opaque client byte stored in `handle`'s entry. Returns 0 if + /// never set (including every chunk created before this feature). Rejects + /// deleted handles with `InvalidHandle` via the shared `lookup_live` guard + /// (I125 — `read`, `tag`, and `delete_tagged` apply the identical rule). + /// Takes `&self`. + pub fn client_byte(&self, handle: u64) -> Result { + self.check_alive()?; + let result = self.client_byte_inner(handle); + self.poison_on_fatal(result) + } + + fn client_byte_inner(&self, handle: u64) -> Result { + Ok(self.lookup_live(handle)?.client_byte) + } + + /// Set the opaque client byte for `handle`. Requires an active + /// transaction; durable on commit, reverted on rollback. Any `u8` is + /// valid. COWs only the handle-table leaf — no data-page, overflow, or + /// membership-index work. Takes `&mut self`. + pub fn set_client_byte(&mut self, handle: u64, byte: u8) -> Result<()> { + self.check_alive()?; + let result = self.set_client_byte_inner(handle, byte); + self.poison_on_fatal(result) + } + + fn set_client_byte_inner(&mut self, handle: u64, byte: u8) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + let mut entry = self.lookup_live(handle)?; + entry.client_byte = byte; + self.ht_insert(handle, &entry)?; + Ok(()) + } + + pub fn handles_with_tag(&self, tag: u32) -> Result> { + self.check_alive()?; + let result = self.handles_with_tag_inner(tag); + self.poison_on_fatal(result) + } + + fn handles_with_tag_inner(&self, tag: u32) -> Result> { + let root = if self.active_txn { + self.current_roots.membership_index_page + } else { + self.committed_roots.membership_index_page + }; + let mut cache = self.cache.borrow_mut(); + // No PAGE_ID_NONE guard (unlike tag_inner): an empty/absent index is a + // legitimate "no handles with this tag" -> handles_for_tag returns an + // empty Vec for a PAGE_ID_NONE root, whereas a missing handle table is an error. + self.membership_index.handles_for_tag(&mut cache, root, tag) + } + + /// Read a value by handle. + /// + /// If a transaction is active, reads see the in-progress (uncommitted) state + /// through current_roots — i.e. "read your own writes". Otherwise reads go + /// through committed_roots, the last durably-committed snapshot. There is no + /// MVCC / snapshot isolation for concurrent readers because the writer is + /// single-threaded; this branch is purely about making the active writer + /// see its own pending mutations. + /// + /// F3: takes `&self`. Internally, the page cache is wrapped in a + /// RefCell so that the mutation required by LRU bookkeeping / page + /// loading can happen behind a shared reference. See the field-level + /// comment on `cache` for the full rationale and why RefCell was + /// chosen over Mutex. + pub fn read(&self, handle: u64) -> Result> { + self.check_alive()?; + let result = self.read_inner(handle); + self.poison_on_fatal(result) + } + + fn read_inner(&self, handle: u64) -> Result> { + let entry = self.lookup_live(handle)?; + + let mut cache = self.cache.borrow_mut(); + match entry.flags { + HandleFlags::Live => { + let buf = cache.get(entry.page_id)?; + // The handle-table entry insists this slot is live. If + // `DataPage::read` returns None anyway, the data page's + // structural state disagrees with the handle table — the + // page header, slot directory, or slot entry is damaged. + // That's CorruptPage (fatal / poisons the manager), not + // InvalidHandle (operational). + match DataPage::read(buf, entry.slot_index) { + Some(data) => Ok(data.to_vec()), + None => Err(ChiselError::CorruptPage { + page_id: entry.page_id, + }), + } + } + HandleFlags::Overflow => Overflow::read(&mut cache, entry.page_id), + // Unreachable in practice: `lookup_live` already excludes tombstones + // (I125). Kept as an exhaustive, non-panicking backstop — if the + // liveness invariant were ever violated, read still returns the + // operational `InvalidHandle` rather than aborting the writer. + HandleFlags::Deleted => Err(ChiselError::InvalidHandle(handle)), + } + } + + /// Iterate over all live handles. + /// + /// F3: takes `&self` (same rationale as `read`). + pub fn handles(&self) -> Result> { + self.check_alive()?; + let result = self.handles_inner(); + self.poison_on_fatal(result) + } + + fn handles_inner(&self) -> Result> { + let root = self.live_handle_table_root(); + if root == PAGE_ID_NONE { + return Ok(Vec::new()); + } + let mut cache = self.cache.borrow_mut(); + let entries = self.handle_table.iter_live(&mut cache, root)?; + Ok(entries.into_iter().map(|(h, _)| h).collect()) + } +} diff --git a/src/transaction/recovery.rs b/src/transaction/recovery.rs new file mode 100644 index 0000000..7d73c06 --- /dev/null +++ b/src/transaction/recovery.rs @@ -0,0 +1,334 @@ +//! transaction::recovery — the two `TransactionManager` constructors: +//! `create_new` (fresh database, staggered superblock bank) and +//! `open_existing` (recover the highest valid superblock). Split out of +//! `transaction.rs` verbatim; see the parent module for the type and fields. + +use super::*; + +impl TransactionManager { + /// Create a new database with `superblock_count` superblock slots. + /// + /// All N slots are initialized as VALID superblocks at staggered + /// counters 0..N-1 (slot i gets counter N-1-i). This matters for + /// crash safety (ISSUES.md I2 + R4): + /// + /// * The I2 fix for N=2: if the first user commit (which writes + /// slot 0 at counter N) is torn, slot 1 at counter N-2 still + /// holds a valid "empty database" superblock so the file stays + /// openable. + /// * The R4 generalization for N>=3: multiple staggered fallback + /// slots survive CONSECUTIVE torn writes. For N=3, slots 1 and 2 + /// both hold valid empty states at lower counters after slot 0 + /// is written; a torn retry of the same commit still has slot 2 + /// to fall back to. + /// + /// An fsync is issued before returning so the whole bank of slots + /// is durable before any user data is written. Slot counters with + /// the value 0 are SAFE even though zero bits are "the natural + /// value of an uninitialized disk region" because `select()` + /// filters on XXH3 checksum validity BEFORE comparing counters — + /// a legitimate counter-0 slot has a valid checksum; a zeroed + /// region doesn't. + pub fn create_new(mut cache: PageCache, superblock_count: u32) -> Result { + // Caller is expected to have validated bounds via Options in + // lib.rs, but defend against direct-call misuse too. + assert!( + (2..=MAX_SUPERBLOCKS).contains(&superblock_count), + "superblock_count {superblock_count} out of supported range 2..=16" + ); + + // Write N staggered slots. Slot 0 gets the highest counter + // (superblock_count - 1), slot N-1 gets 0. First user commit + // bumps to N, which modulo N is 0, so slot 0 is the first to + // be overwritten — the behavior the I2 fix established for + // N=2 generalizes cleanly to larger N. + // + // Invariant after this loop: every slot is a valid superblock + // referencing the same (empty) roots, at counters 0..N-1. + // `select()` at open time will pick slot 0 (highest counter). + // After the first user commit, slot 0 holds the newest data + // and the rest remain as "rollback fallbacks". + let mut sb = Superblock::new_empty(superblock_count); + for i in 0..superblock_count { + sb.txn_counter = (superblock_count - 1 - i) as u64; + let buf = sb.serialize(); + cache.io_mut().write_page(i as u64, &buf)?; + } + cache.io_mut().fsync()?; + cache.set_next_page_id(superblock_count as u64); + + let roots = Roots { + handle_table_page: PAGE_ID_NONE, + // No freemap tree yet: the first allocation falls through to extend + // (nothing to reuse) and the first free materializes the tree lazily + // (persist_freemap calls FreeMapTree::create). Depth 0 matches the + // single-leaf format. + freemap_page: PAGE_ID_NONE, + freemap_depth: 0, + // Start at 1: handle 0 is reserved as the "no handle" sentinel and is + // never minted (see Superblock::new_empty, which seeds the persisted + // superblock the same way). Must match new_empty so the in-memory + // roots and the on-disk superblock of a fresh store agree. + next_handle: 1, + total_pages: superblock_count as u64, + named_roots: [NamedRoot::EMPTY; NAMED_ROOT_COUNT], + membership_index_page: PAGE_ID_NONE, + }; + + Ok(TransactionManager { + cache: RefCell::new(cache), + committed_roots: roots.clone(), + current_roots: roots, + handle_table: HandleTable::new(), + membership_index: MembershipIndex::new(), + // Slot 0 was written last in the loop above, at counter + // (superblock_count - 1 - 0) = superblock_count - 1. That's + // the highest counter and therefore the winner on select(). + txn_counter: (superblock_count - 1) as u64, + superblock_count, + active_txn: false, + savepoints: Vec::new(), + txn_freed_pages: Vec::new(), + freemap_hint: 0, + pending_structural_frees: Vec::new(), + structural_reuse: Vec::new(), + structural_superseded: Vec::new(), + freemap_session_owned: FxHashSet::default(), + // A fresh database has no data pages and no live slots yet. + committed_live_slots: FxHashMap::default(), + current_live_slots: FxHashMap::default(), + insert_cursor: None, + poisoned: Cell::new(false), + #[cfg(test)] + fault: fault::FaultInjector::default(), + }) + } + + /// Open an existing database from file. + /// + /// This is the crash recovery path. All N superblock slots (where + /// N is discovered from disk — see the probe below) are read, + /// `Superblock::select()` picks the one with the highest + /// txn_counter and a valid XXH3 checksum, and a torn write to the + /// most-recently-targeted slot silently falls back to the next + /// best survivor. No log replay required. + /// + /// R4 slot discovery: + /// + /// The number of superblock slots is NOT a compile-time constant. + /// A database created with `superblock_count=4` has 4 slots at + /// pages 0..3; a default database has 2 at pages 0..1. To find N + /// without any external hint we: + /// + /// 1. Read the first MAX_SUPERBLOCKS pages of the file (bounded + /// by EOF — a fresh DB has exactly N pages and no more). We + /// deliberately do NOT short-circuit on "this page doesn't + /// look like a superblock": a torn write that hit the magic + /// bytes of an otherwise-valid slot would look like garbage, + /// and short-circuiting would skip past the legitimate + /// successor slots. Reading a few extra pages is cheap. + /// 2. Pass all candidates to `Superblock::select`, which uses + /// `deserialize` to filter on XXH3 checksum + MAGIC bytes. + /// Data pages that happen to sit at positions < MAX_SUPERBLOCKS + /// (e.g., in a database where N=2 and there's a data page at + /// page 2) fail the magic check and are harmlessly ignored. + /// 3. The winner's `superblock_count` field tells us N, which we + /// cache on the TransactionManager for commit-time slot + /// selection. + /// + /// If no valid superblock is found in the first MAX_SUPERBLOCKS + /// pages, we return `CorruptSuperblock`. This bounds the probe + /// cost in the pathological case where every candidate is torn. + pub fn open_existing(mut cache: PageCache) -> Result { + // Step 1: read up to MAX_SUPERBLOCKS pages as candidates. + let mut candidates: Vec<[u8; PAGE_SIZE]> = Vec::new(); + for i in 0..MAX_SUPERBLOCKS as u64 { + // If the file is shorter than MAX_SUPERBLOCKS (fresh DB + // with small N), read_page returns InvalidPageId (I16). + // Stop probing at EOF. + match cache.io_mut().read_page(i) { + Ok(buf) => candidates.push(buf), + Err(ChiselError::InvalidPageId { .. }) => break, + Err(e) => return Err(e), + } + } + + // Step 2 + 3: pick the winner via select(). select() uses + // deserialize, which validates checksum and magic — data + // pages in the candidate list (if any) are filtered out. + let sb = Superblock::select(&candidates).ok_or_else(|| ChiselError::CorruptSuperblock { + defects: Superblock::diagnose(&candidates), + })?; + + // Format-version gate (see ISSUES.md I15 for the original check, + // I29 for the major/minor split). Compare MAJOR only: the packed + // u32 layout (upper 16 = major, lower 16 = minor) lets same-major + // files open regardless of minor drift, which is what makes the + // README's "sacred within a major version" promise enforceable. + // Minor-newer files are accepted (read-compatible) here but forced + // read-only by the I29 write-gate immediately below — writing them + // would drop fields this binary doesn't know about. + // + // We validate AFTER select() rather than inside deserialize() + // because the winning superblock's version is what determines + // compatibility — silently falling back to an older-version + // superblock would hand the user a stale snapshot with + // mysteriously missing data. + if page::format_major(sb.format_version) != page::FORMAT_MAJOR_VERSION { + return Err(ChiselError::UnsupportedFormatVersion { + found: sb.format_version, + expected: page::FORMAT_VERSION, + }); + } + + // Reject files written with a different page geometry before reading any + // data pages — every page boundary calculation would be wrong if the page + // size differed. The superblock.rs deserialize() reads the field but does + // not validate it (a size mismatch is not a torn-slot signal; it must not + // cause select() to fall back to a sibling slot). This is the right place + // to raise it: after select() has picked the winning slot but before any + // data is touched. + if sb.page_size != PAGE_SIZE as u32 { + return Err(ChiselError::UnsupportedPageSize { + stored: sb.page_size, + compiled: PAGE_SIZE as u32, + }); + } + + // I29 write-gate: a file whose MINOR exceeds this binary's may contain + // version-requiring page layouts we cannot safely write — we would + // stamp pages at our older minor and drop the newer fields. Reads ARE + // safe (within a MAJOR all layout changes are additive, so known fields + // sit at stable offsets), so we open the file but force it read-only; + // mutations then return ReadOnlyMode. The complementary I31 per-page + // read-dispatch lets a newer binary read these older pages. + // See docs/specs/2026-06-21-per-page-format-versioning-design.md. + if page::format_minor(sb.format_version) > page::FORMAT_MINOR_VERSION { + cache.io_mut().force_read_only(); + } + + let page_count = cache.io_mut().page_count()?; + if page_count < sb.total_pages { + return Err(ChiselError::FileSizeMismatch { + // saturating_mul: `sb.total_pages` comes from a checksum-valid but + // otherwise untrusted superblock — `Superblock::deserialize` bounds + // only the checksum, MAGIC, and superblock_count, NOT total_pages. + // A crafted/edited file with total_pages near u64::MAX would + // overflow `* PAGE_SIZE` here: a panic in debug builds (how CI + // runs) and a silent wrap in release. Saturating keeps the public + // `Chisel::open` a typed-error path; "as many bytes as a u64 can + // represent" is the right report for an absurd page count (mirrors + // the I47 saturation in `Chisel::stats`/`file_size_bytes`). + // `page_count` is file-length-bounded and cannot realistically + // overflow, but it is saturated too for symmetry. + expected: sb.total_pages.saturating_mul(PAGE_SIZE as u64), + actual: page_count.saturating_mul(PAGE_SIZE as u64), + }); + } + // Reset next_page_id from the authoritative superblock, NOT from + // the on-disk file length (ISSUES.md I4). This matters because a + // crash mid-rollback could leave the file extended past the + // committed superblock's `total_pages` — those trailing pages are + // unreferenced garbage, and letting `new_page()` allocate above + // them would mean the next commit's new pages live at the very + // end of the file while the garbage sits in the middle. Reseeding + // from `sb.total_pages` causes the next allocations to overwrite + // the garbage, which is exactly what we want. The rollback-path + // truncation added by I3 also prevents this situation from + // arising in the first place, but the reseed is a defense-in- + // depth guarantee against any crash that happened before I3 or + // against external truncation/corruption tools. + cache.set_next_page_id(sb.total_pages); + + let roots = Roots { + handle_table_page: sb.root_handle_table_page, + // The committed freemap tree IS {root, depth} from the superblock — + // no separate in-memory mirror is loaded. Depth 0 (the default for + // pre-multi-page databases) reaches today's single-leaf format. + freemap_page: sb.root_freemap_page, + freemap_depth: sb.freemap_depth, + next_handle: sb.next_handle, + total_pages: sb.total_pages, + named_roots: sb.named_roots, + // Normalize old files (pre-chunk-tags bytes were zeroed) so the + // rest of the engine has a single "empty" sentinel: PAGE_ID_NONE. + membership_index_page: if sb.root_membership_index_page == 0 { + PAGE_ID_NONE + } else { + sb.root_membership_index_page + }, + }; + + // The HandleTable struct keeps only its depth in memory; physical pages + // live in the cache, reached via the root page_id in the superblock. + // Reconstruct depth by walking the left spine (see HandleTable::recover_depth). + let mut ht = HandleTable::new(); + ht.set_depth(HandleTable::recover_depth( + &mut cache, + sb.root_handle_table_page, + )?); + + // Mirror the handle-table depth recovery for the membership index: its + // outer RadixU64 keeps only depth in memory, rebuilt by walking the + // persisted spine from the root recorded in the superblock. Uses the + // normalized roots.membership_index_page (PAGE_ID_NONE for legacy files). + let mut membership_index = MembershipIndex::new(); + if roots.membership_index_page != PAGE_ID_NONE { + let depth = RadixU64::recover_depth(&mut cache, roots.membership_index_page)?; + membership_index.set_outer_depth(depth); + } + + // The committed freemap tree is reconstructed on demand from + // {root_freemap_page, freemap_depth} via FreeMapTree::from_roots — no + // eager in-memory mirror is loaded here. A DB created under v1 (pre-R2) + // or a fresh one has root_freemap_page == PAGE_ID_NONE, which + // from_roots treats as the empty (nothing-free) tree. Tree pages are + // checksum-validated on cache miss as they are descended, so a torn or + // corrupt freemap surfaces as a fatal error rather than silent reuse. + + // Rebuild the live-slot count map (ISSUES.md R1) by scanning the + // handle table. Every Live entry contributes one live slot to + // its target data page; Overflow and Deleted entries don't + // count. Cost is O(live handles), paid once at open. In-memory + // only — the alternative (storing the count on the data page + // itself) would require COWing pages on every delete, which + // shadow paging cannot afford. + let mut committed_live_slots: FxHashMap = FxHashMap::default(); + if sb.root_handle_table_page != PAGE_ID_NONE { + let entries = ht.iter_live(&mut cache, sb.root_handle_table_page)?; + for (_, entry) in entries { + if entry.flags == HandleFlags::Live { + *committed_live_slots.entry(entry.page_id).or_insert(0) += 1; + } + } + } + let current_live_slots = committed_live_slots.clone(); + + Ok(TransactionManager { + cache: RefCell::new(cache), + committed_roots: roots.clone(), + current_roots: roots, + handle_table: ht, + membership_index, + txn_counter: sb.txn_counter, + // R4: discovered from the winning superblock's own + // `superblock_count` field. Cached so commit doesn't have + // to re-look it up for slot selection. + superblock_count: sb.superblock_count, + active_txn: false, + savepoints: Vec::new(), + txn_freed_pages: Vec::new(), + freemap_hint: 0, + pending_structural_frees: Vec::new(), + structural_reuse: Vec::new(), + structural_superseded: Vec::new(), + freemap_session_owned: FxHashSet::default(), + committed_live_slots, + current_live_slots, + insert_cursor: None, + poisoned: Cell::new(false), + #[cfg(test)] + fault: fault::FaultInjector::default(), + }) + } +} diff --git a/src/transaction/savepoints.rs b/src/transaction/savepoints.rs new file mode 100644 index 0000000..7bfbc09 --- /dev/null +++ b/src/transaction/savepoints.rs @@ -0,0 +1,145 @@ +//! transaction::savepoints — nested savepoint scopes: savepoint / +//! rollback_to / release (+ their `_inner` cores). Split out of +//! `transaction.rs` verbatim; see the parent module for the type and fields. + +use super::*; + +impl TransactionManager { + /// Push a named savepoint onto the stack. Captures the current + /// `next_page_id` watermark so `rollback_to(name)` can truncate the + /// cache back to this exact point. `freed_pages` is moved INTO the + /// savepoint record so the enclosing transaction's `txn_freed_pages` + /// accumulates only frees from the savepoint's own scope. + pub fn savepoint(&mut self, name: &str) -> Result<()> { + self.check_alive()?; + let result = self.savepoint_inner(name); + self.poison_on_fatal(result) + } + + fn savepoint_inner(&mut self, name: &str) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + if self.savepoints.iter().any(|sp| sp.name == name) { + return Err(ChiselError::DuplicateSavepoint(name.to_string())); + } + let watermark = self.cache_watermark(); + // R1: snapshot the live-slot map and the cursor. Also drop the + // cursor in the active scope — once a savepoint exists, the + // insert path stops packing into the cursor (same posture as + // freemap reuse: savepoints disable the optimization so the + // rollback_to semantics stay simple). + let live_slots = self.current_live_slots.clone(); + let insert_cursor = self.insert_cursor; + self.insert_cursor = None; + self.savepoints.push(Savepoint { + name: name.to_string(), + roots: self.current_roots.clone(), + watermark, + freed_pages: std::mem::take(&mut self.txn_freed_pages), + live_slots, + insert_cursor, + }); + Ok(()) + } + + /// Roll back to a named savepoint without ending the transaction. + /// Truncates the cache to the savepoint's watermark (discarding every + /// page allocated after the savepoint), restores the roots snapshot, + /// and pops any savepoints layered on top. The named savepoint itself + /// remains on the stack and can be rolled back to again or released. + /// + /// NOTE: `freed_pages` from savepoints layered on top (and from + /// `self.txn_freed_pages`) are dropped here, which is correct — + /// those frees never became durable, and the roots/page contents + /// those frees described have been rewound along with the cache + /// truncate. Post-R2, `commit()` DOES return freed pages to the + /// freemap; this rollback path simply discards the unfinished + /// accounting. + pub fn rollback_to(&mut self, name: &str) -> Result<()> { + self.check_alive()?; + let result = self.rollback_to_inner(name); + self.poison_on_fatal(result) + } + + fn rollback_to_inner(&mut self, name: &str) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + let idx = self + .savepoints + .iter() + .position(|sp| sp.name == name) + .ok_or_else(|| ChiselError::SavepointNotFound(name.to_string()))?; + + let watermark = self.savepoints[idx].watermark; + self.cache.borrow_mut().truncate(watermark)?; + + self.current_roots = self.savepoints[idx].roots.clone(); + // C1: re-derive outer_depth from the restored savepoint root (see rollback_inner). + { + let mut cache = self.cache.borrow_mut(); + let depth = + RadixU64::recover_depth(&mut cache, self.current_roots.membership_index_page)?; + self.membership_index.set_outer_depth(depth); + } + // I99: re-derive handle-table depth from the restored savepoint root + // (same rationale as rollback_inner / outer_depth). + { + let mut cache = self.cache.borrow_mut(); + let depth = + HandleTable::recover_depth(&mut cache, self.current_roots.handle_table_page)?; + self.handle_table.set_depth(depth); + } + // R1: restore live-slot counts and cursor from the savepoint + // snapshot. The cursor was force-cleared when the savepoint was + // created, so this sets the cursor back to whatever value it + // held BEFORE the savepoint was taken (typically also None, + // since savepoint-bearing transactions disable packing). + self.current_live_slots = self.savepoints[idx].live_slots.clone(); + self.insert_cursor = self.savepoints[idx].insert_cursor; + self.savepoints.truncate(idx + 1); + self.txn_freed_pages.clear(); + + Ok(()) + } + + /// Release (flatten) a named savepoint and everything layered on top + /// of it. Under watermark-based rollback, this is just `savepoints + /// .truncate(idx)` plus a merge of freed-page lists — the released + /// savepoints' allocated pages remain reachable via the outer + /// watermark (i.e. `committed_roots.total_pages`), which is still the + /// correct rollback destination for the enclosing transaction. + pub fn release(&mut self, name: &str) -> Result<()> { + self.check_alive()?; + let result = self.release_inner(name); + self.poison_on_fatal(result) + } + + fn release_inner(&mut self, name: &str) -> Result<()> { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + let idx = self + .savepoints + .iter() + .position(|sp| sp.name == name) + .ok_or_else(|| ChiselError::SavepointNotFound(name.to_string()))?; + + // Merge freed_pages from all released savepoints back into the + // current transaction's list. This preserves the invariant that + // txn_freed_pages holds every "frees that would go to the freemap + // on commit" across the entire enclosing transaction, so a later + // rollback correctly drops them. + let mut merged_freed = Vec::new(); + for sp in self.savepoints[idx..].iter() { + merged_freed.extend_from_slice(&sp.freed_pages); + } + merged_freed.append(&mut self.txn_freed_pages); + + self.savepoints.truncate(idx); + self.txn_freed_pages = merged_freed; + + Ok(()) + } +} diff --git a/src/transaction/staging.rs b/src/transaction/staging.rs new file mode 100644 index 0000000..d093549 --- /dev/null +++ b/src/transaction/staging.rs @@ -0,0 +1,342 @@ +//! transaction::staging — atomic allocate staging (I18 / BUG#2): the +//! candidate-prepare / install split for handle-table + membership-index +//! roots, the abort-prepare unwind, and the membership-failure injection +//! hook. Split out of `transaction.rs` verbatim; see the parent module. + +use super::freemap::cow_alloc; +use super::*; + +impl TransactionManager { + /// Insert a value and return a stable handle. + /// + /// Handles are dense u64s drawn from current_roots.next_handle, starting at + /// 1 — handle 0 is reserved as the "no handle" sentinel and is never + /// returned. Large values + /// (> MAX_INLINE_VALUE) go to an overflow chain and the HandleEntry records + /// the first overflow page directly; small values get a slot in a freshly + /// allocated data page. Either way the handle_table.insert() COWs the spine + /// from leaf to root and returns the new root page_id, which becomes the + /// new current_roots.handle_table_page. This is the fundamental shadow- + /// paging step: the old root is still reachable via committed_roots and is + /// untouched on disk until commit swaps the superblock. + pub fn allocate(&mut self, value: &[u8]) -> Result { + self.check_alive()?; + let result = self.allocate_inner(value, 0); + self.poison_on_fatal(result) + } + + pub fn allocate_tagged(&mut self, value: &[u8], tag: u32) -> Result { + self.check_alive()?; + let result = self.allocate_inner(value, tag); + self.poison_on_fatal(result) + } + + /// Compute the membership-index root produced by inserting `(tag, handle)` + /// WITHOUT installing it into `current_roots`; superseded pages are appended + /// to `freed`. Split out so `allocate_inner` can stage the forward- and + /// reverse-map updates and install them atomically (BUG#2). On error, + /// `MembershipIndex::insert` leaves `self.outer_depth` unchanged (it writes + /// back only on success), so the caller need not restore any reverse-map + /// in-memory depth. + fn membership_insert_candidate( + &mut self, + tag: u32, + handle: u64, + freed: &mut Vec, + ) -> Result { + let reuse = self.savepoints.is_empty(); + let mut tree = self.take_freemap_tree(); + let result = { + let hint = &mut self.freemap_hint; + let pool = &mut self.structural_reuse; + let mut cache = self.cache.borrow_mut(); + let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); + self.membership_index.insert( + &mut cache, + self.current_roots.membership_index_page, + tag, + handle, + &mut alloc, + freed, + ) + }; + // Freemap growth is installed into roots regardless of success: the tree + // pages were extended (never freed), so a non-fatal failure that discards + // `freed`/the candidate root leaves these extra pages as harmless + // above-watermark scratch, exactly like the other COW pages on an aborted + // prepare. put_freemap_tree drains the freemap COW supersedes into + // structural_superseded and returns the session set so the next site in + // this transaction stays in-place. + self.put_freemap_tree(tree); + result + } + + /// Compute the handle-table root produced by inserting `entry` for `handle` + /// WITHOUT installing it into `current_roots`; superseded spine pages are + /// appended to `freed`. The forward-map counterpart to + /// `membership_insert_candidate` for `allocate_inner`'s atomic staging + /// (BUG#2). NOTE: `HandleTable::insert` may `grow`, which bumps the + /// in-memory descent depth eagerly — the caller captures and restores that + /// depth on the prepare-abort path. + pub(super) fn handle_table_insert_candidate( + &mut self, + handle: u64, + entry: &HandleEntry, + freed: &mut Vec, + ) -> Result { + // Test-only injection (see `fail_next_handle_table_op`): simulate a + // non-fatal CacheFull at the forward / handle-table step — the one + // carrying the eager depth bump for allocate. Lives here so BOTH + // allocate_inner and update_inner exercise the real abort/unwind. No + // production artifact under `#[cfg(not(test))]`. + #[cfg(test)] + if self.fault.fail_next_handle_table_op.replace(false) { + return Err(ChiselError::CacheFull { limit: 0 }); + } + let reuse = self.savepoints.is_empty(); + let mut tree = self.take_freemap_tree(); + let result = { + let hint = &mut self.freemap_hint; + let pool = &mut self.structural_reuse; + let mut cache = self.cache.borrow_mut(); + let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); + self.handle_table.insert( + &mut cache, + self.current_roots.handle_table_page, + handle, + entry, + &mut alloc, + freed, + ) + }; + // Install freemap growth into roots (and return the session set) so the + // NEXT candidate in this allocate (the reverse-map insert) threads the + // up-to-date tree and treats already-COW'd freemap pages as in-place. The + // freemap COW supersedes go to structural_superseded via put_freemap_tree. + // See membership_insert_candidate for the abort-safety reasoning. + self.put_freemap_tree(tree); + result + } + + /// Unwind the installed state from a partially-completed `allocate_inner` + /// PREPARE phase after a non-fatal failure. + /// + /// WHAT IS RESTORED (the installed state is a no-op): + /// - `current_roots.handle_table_page` — reverts to the pre-allocate value, + /// undoing any lazy `ensure_handle_table` materialization (empty DB goes + /// back to `PAGE_ID_NONE`). + /// - `handle_table` descent depth — restored from the saved value, undoing + /// the eager bump that `HandleTable::grow` applies before its fallible COW. + /// - Inline value's data slot — released via `release_data_slot` so + /// `current_live_slots` and the insert cursor stay consistent with the + /// un-installed root. A page allocated solely for this value goes to zero + /// occupancy and is queued for reclamation; a shared cursor page keeps a + /// defrag-reclaimable dead slot (exactly like a normal delete). + /// - `next_handle` — never consumed (bumped only in the infallible INSTALL + /// phase), so there is nothing to undo here. + /// + /// WHAT IS NOT RESTORED (a bounded allocated-but-unreferenced residue): + /// - Freemap COW pages drawn during the candidate allocations. When reuse is + /// enabled, `cow_alloc` may have cleared free bits in the committed freemap + /// and advanced the tree to cover the candidate-spine pages, leaving those + /// ids allocated-but-unreferenced. Restoring them here would require either + /// re-marking them free (fighting the COW dirty-page I20 invariant) or + /// re-sorting and re-queuing them as freed data pages (introducing I20 + /// dirty-page hazards and growth regressions on the abnormal path). + /// Instead, the residue is reclaimed by the expected rollback + /// (`discard_all_dirty` + watermark truncate restore the freemap to its + /// committed state). Overflow value pages are in the same residue class. + /// The residue leaks only if the caller commits after the operational + /// error rather than rolling back — contrary to documented contract. + fn abort_allocate_prepare( + &mut self, + saved_root: u64, + saved_depth: u32, + inline_page: Option, + ) { + self.current_roots.handle_table_page = saved_root; + self.handle_table.set_depth(saved_depth); + if let Some(page_id) = inline_page { + self.release_data_slot(page_id); + } + } + + /// Compute the membership-index root produced by removing `(tag, handle)` + /// WITHOUT installing it; returns `(new_root, was_present)`. Counterpart to + /// `membership_insert_candidate` for `delete_inner`'s atomic staging (BUG#2). + pub(super) fn membership_remove_candidate( + &mut self, + tag: u32, + handle: u64, + freed: &mut Vec, + ) -> Result<(u64, bool)> { + let reuse = self.savepoints.is_empty(); + let mut tree = self.take_freemap_tree(); + let result = { + let hint = &mut self.freemap_hint; + let pool = &mut self.structural_reuse; + let mut cache = self.cache.borrow_mut(); + let mut alloc = |c: &mut PageCache| cow_alloc(c, &mut tree, hint, pool, reuse); + self.membership_index.remove( + &mut cache, + self.current_roots.membership_index_page, + tag, + handle, + &mut alloc, + freed, + ) + }; + self.put_freemap_tree(tree); + result + } + + /// Test-only fault decision for the reverse-map (membership-index) step, + /// shared by `allocate_inner` and `delete_inner`. Returns true (inject a + /// non-fatal CacheFull) if the one-shot `fail_next_membership_op` is armed, + /// or if the `fail_membership_op_after` countdown reaches this op. Consuming + /// here keeps the injection logic in one place. + #[cfg(test)] + pub(super) fn inject_membership_failure(&self) -> bool { + if self.fault.fail_next_membership_op.replace(false) { + return true; + } + let remaining = self.fault.fail_membership_op_after.get(); + if remaining == 0 { + return false; + } + self.fault.fail_membership_op_after.set(remaining - 1); + remaining == 1 + } + + fn allocate_inner(&mut self, value: &[u8], tag: u32) -> Result { + if !self.active_txn { + return Err(ChiselError::NoActiveTransaction); + } + + // BUG#2 atomic staging: the FORWARD map (the chunk's HandleEntry.tag in + // the handle table) and the REVERSE map (tag -> handles in the + // membership index, powering handles_with_tag / delete-by-tag) must + // become durable together. We compute both candidate roots in a fallible + // PREPARE phase that never installs into `current_roots`, then install + // them together in an infallible phase. + // + // A non-fatal CacheFull/SpillwayFull mid-prepare is unwound by + // `abort_allocate_prepare`, which is a no-op for the INSTALLED state: + // neither forward nor reverse map changes, the eagerly-bumped + // handle-table depth and any lazily-created root are restored, the + // inline value's data slot is released (keeping live-slot / cursor + // accounting consistent), and the handle id is not consumed + // (next_handle is bumped only on success). + // + // What the abort does NOT restore is the freemap COW that the candidate + // allocations performed: when reuse is enabled, `cow_alloc` may have + // drawn candidate-spine pages from the freemap (clearing their bits and + // advancing the tree), and those page ids are now allocated-but- + // unreferenced. This is a BOUNDED residue — the same class as any + // post-allocation failure — fully reclaimed by the expected rollback + // (discard_all_dirty + watermark truncate restore the freemap to its + // committed state). It materializes as a leak only if the caller commits + // after the operational error instead of rolling back, which is contrary + // to the documented contract. + let handle = self.current_roots.next_handle; + + // Value storage (PREPARE). For an inline value this also bumps + // current_live_slots and may set the insert cursor; capture its page id + // so a later prepare failure can release it via `abort_allocate_prepare`, + // keeping that bookkeeping consistent with the un-installed root. + // Overflow storage has no live-slot/cursor side effects. + let mut inline_page: Option = None; + let entry = if value.len() > MAX_INLINE_VALUE { + let first_page = { + let mut cache = self.cache.borrow_mut(); + Overflow::write(&mut cache, value)? + }; + HandleEntry { + page_id: first_page, + slot_index: 0, + flags: HandleFlags::Overflow, + tag, + client_byte: 0, + } + } else { + let (data_page_id, slot) = self.insert_into_data_page(value)?; + inline_page = Some(data_page_id); + HandleEntry { + page_id: data_page_id, + slot_index: slot, + flags: HandleFlags::Live, + tag, + client_byte: 0, + } + }; + + // Capture the handle-table root/depth BEFORE `ensure_handle_table` may + // lazily materialize an empty root, so a prepare failure restores + // current_roots to its true pre-allocate state (an empty DB reverts to + // PAGE_ID_NONE). The depth capture also covers `HandleTable::grow`, which + // bumps the in-memory descent depth EAGERLY (before its fallible leaf + // COW) while we defer the root install. The membership index needs no + // such save: MembershipIndex::insert writes its outer_depth back only on + // the success path, so a failed reverse-map op never advances it. + let saved_ht_root = self.current_roots.handle_table_page; + let saved_ht_depth = self.handle_table.depth(); + self.ensure_handle_table()?; + + // FORWARD map: compute the new handle-table root; do NOT install yet. + // (handle_table_insert_candidate carries the #[cfg(test)] forward-step + // fault injection, shared with update_inner's handle-table step.) + let mut ht_freed: Vec = Vec::new(); + let ht_new_root = match self.handle_table_insert_candidate(handle, &entry, &mut ht_freed) { + Ok(r) => r, + Err(e) => { + self.abort_allocate_prepare(saved_ht_root, saved_ht_depth, inline_page); + return Err(e); + } + }; + + // REVERSE map: compute the new membership-index root; do NOT install + // yet. tag 0 = untagged (never indexed). + let mut mi_new_root: Option = None; + let mut mi_freed: Vec = Vec::new(); + if tag != 0 { + // Test-only injection (see `fail_next_membership_op`): simulate a + // non-fatal CacheFull at the reverse-map step so the regression test + // exercises the REAL failure handling below. No production artifact: + // the non-test `let res` is the only one compiled outside tests. + #[cfg(test)] + let res: Result = if self.inject_membership_failure() { + Err(ChiselError::CacheFull { limit: 0 }) + } else { + self.membership_insert_candidate(tag, handle, &mut mi_freed) + }; + #[cfg(not(test))] + let res: Result = self.membership_insert_candidate(tag, handle, &mut mi_freed); + + match res { + Ok(r) => mi_new_root = Some(r), + Err(e) => { + // The handle-table insert above already succeeded and may + // have grown the tree (bumping the in-memory depth) and + // produced a candidate root we are now discarding. Unwind the + // whole prepare so current_roots, the depth, and the inline + // value's slot all return to their pre-allocate state. + self.abort_allocate_prepare(saved_ht_root, saved_ht_depth, inline_page); + return Err(e); + } + } + } + + // INSTALL phase (infallible): both maps move together, and only now is + // the handle id consumed and the superseded spine pages queued for + // reclamation at commit. + self.current_roots.next_handle += 1; + self.current_roots.handle_table_page = ht_new_root; + self.txn_freed_pages.append(&mut ht_freed); + if let Some(root) = mi_new_root { + self.current_roots.membership_index_page = root; + self.txn_freed_pages.append(&mut mi_freed); + } + + Ok(handle) + } +} diff --git a/src/transaction/stats.rs b/src/transaction/stats.rs new file mode 100644 index 0000000..ea41e76 --- /dev/null +++ b/src/transaction/stats.rs @@ -0,0 +1,203 @@ +//! transaction::stats — read-only stats / introspection and the +//! defrag-support surface, plus the test-only corruption forges. Split out +//! of `transaction.rs` verbatim; see the parent module for the type and fields. + +use super::*; + +impl TransactionManager { + /// Test-only: forge a freemap orphan exactly as a crash would leave one. + /// Extend a fresh page, stamp it as a checksum-valid `FreeMapInterior`, and + /// return its id WITHOUT referencing it from the live tree or marking it free + /// — the precise state of a structural-recycle-pool page stranded when an + /// in-memory pool is lost to a crash. The orphan sweep + /// (`reclaim_freemap_orphans`) must reclaim it. Returns the forged page id. + /// + /// FreeMapInterior (not FreeMap) is used deliberately: it cannot be mistaken + /// for a freed-bit leaf, and it exercises the interior arm of the type test. + #[cfg(test)] + pub(crate) fn test_forge_freemap_orphan(&mut self) -> Result { + let mut cache = self.cache.borrow_mut(); + let id = cache.new_page()?; + let buf = cache.get_mut(id)?; + buf.fill(0); + buf[0] = crate::page::PageType::FreeMapInterior as u8; + buf[1] = page::current_version(crate::page::PageType::FreeMapInterior); + page::stamp_checksum(buf); + Ok(id) + } + + /// Test-only: forge a CORRUPT, non-reachable page on disk. Extend a fresh + /// page, fill it with garbage, deliberately do NOT stamp a valid checksum, + /// flush it to the backing file, then drop it from the cache so a later + /// `get(id)` re-reads it from disk and fails with `ChecksumMismatch`. The + /// page is never referenced from any tree, so it is a corrupt DEAD page — + /// exactly what the orphan sweep must SKIP rather than poison on. Returns the + /// forged page id. + #[cfg(test)] + pub(crate) fn test_forge_corrupt_dead_page(&mut self) -> Result { + let mut cache = self.cache.borrow_mut(); + let id = cache.new_page()?; + let buf = cache.get_mut(id)?; + // Garbage bytes with a freemap-ish type byte but a checksum that will not + // verify (we never call stamp_checksum). The type byte is irrelevant — + // the read fails the checksum gate before the type is ever inspected. + buf.fill(0xAB); + buf[0] = crate::page::PageType::FreeMap as u8; + cache.flush()?; // write the garbage bytes to the main file + cache.test_drop_from_cache(id); // force a disk re-read (and checksum check) next get + Ok(id) + } + + /// Snapshot the four engine-activity counters (cache hits/misses, + /// pages allocated, fsync calls). Counters are cumulative from the + /// most recent open; the bench harness reads-subtract-reads for + /// per-cell deltas. Takes `&self` (F3); poison-aware via + /// `check_alive`. + pub fn counters(&self) -> Result { + self.check_alive()?; + Ok(self.cache.borrow().counters()) + } + + /// I74 (ISSUES.md, 2026-05-22): peek the spillway's current + /// (logical_bytes, max_bytes) for `Chisel::stats`. `None` if the + /// spillway has never been opened. Routes through the same + /// poison check as `counters()` so a fatal-error state surfaces + /// here too — operators reading `stats()` get a `Poisoned` + /// error rather than stale-looking Some(0,0). + pub fn spillway_capacity(&self) -> Result> { + self.check_alive()?; + Ok(self.cache.borrow().spillway_capacity()) + } + + /// Poisoning-aware wrapper around `PageCache::file_page_count`. Called + /// by `Chisel::stats()` so that a fatal I/O error while measuring the + /// file size also poisons the manager. + /// + /// F3: takes `&self`. + pub fn file_page_count(&self) -> Result { + self.check_alive()?; + let result = self.cache.borrow_mut().file_page_count(); + self.poison_on_fatal(result) + } + + // --- Selective defragmentation support (ISSUES.md R3 + I17) --- + // + // These methods expose just enough of the R1 live-slot tracking + // for `defrag::defrag` to do selective page compaction. The + // defrag module is in-crate and could in principle access the + // fields directly, but going through named methods keeps the + // intent obvious at each call site. + + /// Page ids of data pages whose effective density (live slots / + /// stored slots) is strictly less than `threshold_ratio`. A + /// freshly-packed page with every slot still live has density + /// 1.0; a page that originally packed 39 values but now has only + /// 5 live (34 dead-weight tombstones) has density 0.128 and is a + /// strong defrag candidate. + /// + /// The metric uses the page's OWN stored-slot count (read from + /// the on-disk header via `DataPage::slot_count`) as the + /// denominator — not the max-observed count in the database — + /// because dead-weight slots are what defrag is trying to reclaim. + /// The older "relative to densest" metric failed for the case of a + /// single remaining sparse page (density 1.0 against itself). + /// + /// Returns an empty set when `threshold_ratio <= 0`. Fallible + /// because the per-page stored count is read through the cache. + pub fn sparse_data_pages( + &self, + threshold_ratio: f64, + ) -> Result> { + self.check_alive()?; + let result = self.sparse_data_pages_inner(threshold_ratio); + self.poison_on_fatal(result) + } + + fn sparse_data_pages_inner( + &self, + threshold_ratio: f64, + ) -> Result> { + let mut sparse = std::collections::HashSet::new(); + if threshold_ratio <= 0.0 { + return Ok(sparse); + } + let page_ids: Vec = self.current_live_slots.keys().copied().collect(); + for page_id in page_ids { + let live = match self.current_live_slots.get(&page_id) { + Some(&n) if n > 0 => n, + _ => continue, + }; + let stored = { + let mut cache = self.cache.borrow_mut(); + DataPage::slot_count(cache.get(page_id)?) as u32 + }; + if stored == 0 { + continue; + } + let density = live as f64 / stored as f64; + if density < threshold_ratio { + sparse.insert(page_id); + } + } + Ok(sparse) + } + + /// Snapshot of the page ids currently tracked as holding at least + /// one live slot. Used by `defrag::defrag` for the I17 stat: after + /// the sweep, `pages_freed` is the count of ids that were in this + /// snapshot and are no longer in `current_live_slots` — i.e., + /// pages that the sweep fully drained and returned to the freemap. + /// Net change in the live data-page count is the wrong metric here + /// because a relocation simultaneously drains a sparse page and + /// creates a dense one; the former should count as "reclaimed" + /// even when the latter offsets the net count. + pub fn data_page_ids_snapshot(&self) -> std::collections::HashSet { + self.current_live_slots.keys().copied().collect() + } + + /// Look up the data page id that currently holds `handle`. Returns + /// `Ok(None)` if the handle doesn't exist, is deleted, or points + /// at an overflow chain (for which the notion of "data page" does + /// not apply). + /// + /// Takes `&self`; uses the RefCell around the cache to perform the + /// handle-table lookup. Poisons the manager on fatal I/O or + /// checksum errors. + pub fn handle_live_page_id(&self, handle: u64) -> Result> { + self.check_alive()?; + let result = self.handle_live_page_id_inner(handle); + self.poison_on_fatal(result) + } + + fn handle_live_page_id_inner(&self, handle: u64) -> Result> { + let root = self.live_handle_table_root(); + if root == PAGE_ID_NONE { + return Ok(None); + } + let mut cache = self.cache.borrow_mut(); + let entry = match self.handle_table.lookup(&mut cache, root, handle)? { + Some(e) => e, + None => return Ok(None), + }; + if entry.flags == HandleFlags::Live { + Ok(Some(entry.page_id)) + } else { + Ok(None) + } + } + + /// The handle-table root page id of the active transaction's + /// in-progress roots. Used by `defrag::defrag` to short-circuit + /// the empty-database fast path. + /// + /// I39 (ISSUES.md, 2026-05-22): replaces a `pub fn current_roots() + /// -> (u64, u64, u64)` tuple return that exposed three fields when + /// only one was ever read. YAGNI: if a future caller wants + /// `freemap_page` or `next_handle`, add a sibling accessor at that + /// time rather than guessing the API shape now. `pub(crate)` + /// because `defrag` is the sole intended caller (transaction + /// module became `pub(crate)` in I35). + pub(crate) fn current_handle_table_root_page(&self) -> u64 { + self.current_roots.handle_table_page + } +} diff --git a/src/transaction/tests.rs b/src/transaction/tests.rs new file mode 100644 index 0000000..16f344b --- /dev/null +++ b/src/transaction/tests.rs @@ -0,0 +1,2460 @@ +// transaction::tests — the full unit-test suite for TransactionManager. +// Moved verbatim out of transaction.rs as part of the by-concern module +// split; kept as a single file (the tests share fixtures like fresh_manager +// and assert_no_reachable_page_is_free). + +use super::freemap::take_structural_reuse_log; +use super::*; +use crate::page_io::{Fault, PageIo}; +use tempfile::{NamedTempFile, TempDir}; + +fn fresh_manager() -> TransactionManager { + let file = NamedTempFile::new().unwrap(); + let io = PageIo::open(file.path(), false).unwrap(); + // Match Options::default()'s cache_max_bytes of 8 MiB (1024 pages) + // so tests that intentionally allocate many pages in a single + // transaction (e.g. the I3+I7 handle-table-growth test allocates + // 510+) stay well under the strict cache cap. spillway_max_bytes=0 + // preserves the legacy CacheFull-at-cap behavior in tests. + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let mut tm = TransactionManager::create_new(cache, 2).unwrap(); + // Commit once so there's a real baseline to read/write against. + tm.begin().unwrap(); + tm.commit().unwrap(); + tm +} + +/// C1 invariant: after a commit, NO page reachable from `committed_roots` +/// may be marked free in `committed_freemap`. A correct COW frees only +/// superseded pages; freeing a still-referenced page (the textbook C1 +/// violation — e.g. `grow` freeing the reparented old root, or `update` +/// freeing the OLD value before the new entry is installed) shows up here as +/// a page that is both reachable and free. This is deterministic regardless +/// of the freemap's lowest-id-first selection order, which makes black-box +/// reopen tests unreliable for catching C1. +/// +/// Reachability covers BOTH the index spines (handle-table + membership +/// outer/inner) AND the value storage every live handle points at (its +/// inline data page or its full overflow chain). The value-storage half is +/// essential: a spine-only walk cannot catch a value-page premature-free. +/// +/// PRECONDITION: call only between transactions (right after a commit), where +/// `committed_roots == current_roots` and the in-memory `handle_table.depth` +/// matches the committed root. `iter_live` descends with that live depth, so +/// calling this mid-transaction after a grow would mis-descend. +fn assert_no_reachable_page_is_free(tm: &TransactionManager) { + let mut reachable = Vec::new(); + { + let mut cache = tm.cache.borrow_mut(); + tm.handle_table + .collect_page_ids( + &mut cache, + tm.committed_roots.handle_table_page, + &mut reachable, + ) + .unwrap(); + tm.membership_index + .collect_page_ids( + &mut cache, + tm.committed_roots.membership_index_page, + &mut reachable, + ) + .unwrap(); + + // Value storage reachable through each live HandleEntry. + if tm.committed_roots.handle_table_page != PAGE_ID_NONE { + let live = tm + .handle_table + .iter_live(&mut cache, tm.committed_roots.handle_table_page) + .unwrap(); + for (_handle, entry) in live { + match entry.flags { + HandleFlags::Live => reachable.push(entry.page_id), + HandleFlags::Overflow => { + let chain = + Overflow::collect_chain_pages(&mut cache, entry.page_id).unwrap(); + reachable.extend(chain); + } + HandleFlags::Deleted => {} + } + } + } + } + // Query freeness through the committed freemap TREE (reconstructed from + // {root, depth}) rather than a flat in-memory bitmap — same C1 invariant, + // new storage representation. + let mut cache = tm.cache.borrow_mut(); + let tree = FreeMapTree::from_roots( + tm.committed_roots.freemap_page, + tm.committed_roots.freemap_depth, + ); + for id in reachable { + assert!( + !tree.is_free(&mut cache, id).unwrap(), + "page {id} is reachable from committed_roots but marked FREE in \ + the committed freemap — a still-referenced page was freed (C1 violation)" + ); + } +} + +// COW page reclamation must never free a page still referenced by the +// committed tree, even after the trees GROW (the reparenting paths). Forces +// a handle-table grow (>510 handles) and a membership inner-tree grow +// (>1021 members under one tag), then churns with reclamation, asserting the +// C1 invariant after every commit. +#[test] +fn reclamation_never_frees_a_reachable_page_after_grow() { + let mut tm = fresh_manager(); + let tag = 9u32; + let mut handles = Vec::new(); + let mut v: u32 = 0; + + // Build >1021 tagged members in small batches (stay under the 1024-page + // cache cap), forcing both trees to grow to depth >= 1. + for _ in 0..12 { + tm.begin().unwrap(); + for _ in 0..100 { + let h = tm.allocate_tagged(&v.to_le_bytes(), tag).unwrap(); + handles.push(h); + v += 1; + } + tm.commit().unwrap(); + assert_no_reachable_page_is_free(&tm); + } + assert!( + handles.len() > 1021, + "workload must exceed one membership leaf to force an inner grow" + ); + + // Churn with reclamation across committed transactions: update relocates + // the value and COWs the handle-table spine (freeing the old spine); + // set_client_byte COWs only the leaf. Batched per ~100 handles so a + // single transaction's dirty COW pages stay under the cache cap (within + // a txn, this-txn frees are not yet reusable). Re-check after each commit. + for round in 0..12u32 { + for (chunk_idx, chunk) in handles.chunks(100).enumerate() { + tm.begin().unwrap(); + for (j, h) in chunk.iter().enumerate() { + if (round as usize + chunk_idx + j) % 2 == 0 { + tm.set_client_byte(*h, round as u8).unwrap(); + } else { + tm.update(*h, &round.to_le_bytes()).unwrap(); + } + } + tm.commit().unwrap(); + assert_no_reachable_page_is_free(&tm); + } + } + + // Every handle still carries its tag and is enumerable after the churn. + for h in &handles { + assert_eq!(tm.tag(*h).unwrap(), tag); + } + assert_eq!(tm.handles_with_tag(tag).unwrap().len(), handles.len()); +} + +#[test] +fn tagged_membership_survives_rolled_back_outer_grow() { + let mut tm = fresh_manager(); + // Commit a small tag; the outer (tag-keyed) tree stays depth 0. + tm.begin().unwrap(); + let h = tm.allocate_tagged(b"keep", 3).unwrap(); + tm.commit().unwrap(); + assert_eq!(tm.handles_with_tag(3).unwrap(), vec![h]); + // New txn: a tag >= 1021 forces the outer tree to grow (depth 0 -> 1). + // Roll back. The grown root is discarded and current_roots snaps back to + // the depth-0 committed root; outer_depth must be restored to match. + tm.begin().unwrap(); + let _ = tm.allocate_tagged(b"discard", 5000).unwrap(); + tm.rollback().unwrap(); + // The committed small tag must still be readable (was silently lost before the fix). + assert_eq!( + tm.handles_with_tag(3).unwrap(), + vec![h], + "rolled-back outer grow corrupted committed membership" + ); + assert_eq!(tm.tag(h).unwrap(), 3); + // The discarded tag is gone. + assert_eq!(tm.handles_with_tag(5000).unwrap(), Vec::::new()); +} + +#[test] +fn tagged_membership_survives_rollback_to_savepoint() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let h = tm.allocate_tagged(b"keep", 7).unwrap(); + tm.savepoint("sp").unwrap(); + // Grow the outer tree past depth 0 inside the savepoint, then roll back to it. + let _ = tm.allocate_tagged(b"discard", 6000).unwrap(); + tm.rollback_to("sp").unwrap(); + // Still inside the active txn: the pre-savepoint tag must remain readable. + assert_eq!(tm.handles_with_tag(7).unwrap(), vec![h]); + assert_eq!(tm.handles_with_tag(6000).unwrap(), Vec::::new()); + tm.commit().unwrap(); + assert_eq!(tm.handles_with_tag(7).unwrap(), vec![h]); +} + +// Regression test for ISSUES.md I1. Once the manager is poisoned, +// every public entry point must return ChiselError::Poisoned rather +// than attempting the operation. This is the core invariant of the +// poison model — the test asserts it for each method independently +// so a future refactor that forgets to wrap a new entry point will +// fail loudly. +#[test] +fn poisoned_manager_rejects_every_public_entry_point() { + let mut tm = fresh_manager(); + tm.force_poison_for_test(); + assert!(tm.is_poisoned()); + + assert!(matches!(tm.begin(), Err(ChiselError::Poisoned))); + assert!(matches!(tm.commit(), Err(ChiselError::Poisoned))); + assert!(matches!(tm.rollback(), Err(ChiselError::Poisoned))); + assert!(matches!(tm.savepoint("x"), Err(ChiselError::Poisoned))); + assert!(matches!(tm.rollback_to("x"), Err(ChiselError::Poisoned))); + assert!(matches!(tm.release("x"), Err(ChiselError::Poisoned))); + assert!(matches!(tm.allocate(b"v"), Err(ChiselError::Poisoned))); + assert!(matches!(tm.read(0), Err(ChiselError::Poisoned))); + assert!(matches!(tm.update(0, b"v"), Err(ChiselError::Poisoned))); + assert!(matches!(tm.delete(0), Err(ChiselError::Poisoned))); + assert!(matches!(tm.handles(), Err(ChiselError::Poisoned))); + assert!(matches!(tm.file_page_count(), Err(ChiselError::Poisoned))); + assert!(matches!( + tm.allocate_tagged(b"v", 1), + Err(ChiselError::Poisoned) + )); + assert!(matches!(tm.tag(0), Err(ChiselError::Poisoned))); + assert!(matches!(tm.handles_with_tag(1), Err(ChiselError::Poisoned))); + assert!(matches!(tm.delete_tagged(0, 1), Err(ChiselError::Poisoned))); + assert!(matches!( + tm.delete_with_tag(1, 10), + Err(ChiselError::Poisoned) + )); + assert!(matches!(tm.client_byte(0), Err(ChiselError::Poisoned))); + assert!(matches!( + tm.set_client_byte(0, 1), + Err(ChiselError::Poisoned) + )); +} + +#[test] +fn fatal_error_outside_commit_also_poisons() { + // I112: a REAL fatal IoError on a cold read OUTSIDE any transaction + // poisons the manager (the non-commit fatal path, poison_on_fatal). This + // replaces the old force_poison_for_test() tautology with an injected + // fault. We reopen over the committed file so read(h) is a cache MISS + // that actually reaches read_page(pid). + let file = NamedTempFile::new().unwrap(); + let h; + let pid; + { + let io = PageIo::open(file.path(), false).unwrap(); + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let mut tm = TransactionManager::create_new(cache, 2).unwrap(); + tm.begin().unwrap(); + h = tm.allocate(b"durable").unwrap(); + tm.commit().unwrap(); + pid = tm.handle_live_page_id(h).unwrap().expect("live data page"); + } + + // Reopen: cold cache, so read(h) misses and calls read_page(pid). + let io = PageIo::open(file.path(), false).unwrap(); + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let tm = TransactionManager::open_existing(cache).unwrap(); + tm.cache.borrow().io().arm_fault(Fault::FailReadPage(pid)); + let result = tm.read(h); + assert!( + matches!(result, Err(ChiselError::IoError(_))), + "cold read fault must surface IoError, got {result:?}" + ); + assert!( + tm.is_poisoned(), + "a fatal read error outside commit must poison" + ); +} + +// Regression test for ISSUES.md I3 + I7. A transaction that forces +// handle-table growth allocates many pages (the data pages for each +// value, the handle-table leaves, the COW spine clones, and the +// new interior root from grow()). After rollback, every one of those +// pages must be gone — both from the in-memory cache AND from the +// file itself. +// +// Pre-I7, the old per-page dirty list missed intermediate COW pages. +// Pre-I3, rollback only discarded cache entries without truncating +// the file, so the extended pages leaked permanently. This test +// exercises both conditions in one shot by asserting the +// `next_page_id` watermark and the cache page-count return to their +// pre-transaction values after rollback. +#[test] +fn rollback_truncates_cache_and_file_to_pre_txn_watermark() { + let mut tm = fresh_manager(); + let pre_watermark = tm.cache.borrow().next_page_id(); + let pre_file_pages = tm.cache.borrow_mut().file_page_count().unwrap(); + + tm.begin().unwrap(); + tm.allocate(b"seed").unwrap(); + // Force handle-table growth by crossing the 510-entry leaf boundary. + for _ in 0..510 { + tm.allocate(b"f").unwrap(); + } + // Sanity: the transaction must have extended the cache past the + // pre-transaction watermark. Otherwise the test below is vacuous. + let mid_watermark = tm.cache.borrow().next_page_id(); + assert!( + mid_watermark > pre_watermark + 510, + "expected the transaction to allocate many pages beyond {pre_watermark}, got {mid_watermark}" + ); + + tm.rollback().unwrap(); + + let post_watermark = tm.cache.borrow().next_page_id(); + let post_file_pages = tm.cache.borrow_mut().file_page_count().unwrap(); + assert_eq!( + post_watermark, pre_watermark, + "rollback must rewind next_page_id to the pre-transaction watermark" + ); + assert_eq!( + post_file_pages, pre_file_pages, + "rollback must truncate the file back to its pre-transaction page count" + ); +} + +// rollback_to(name) must truncate cache+file to the savepoint's +// watermark, discarding every page allocated after the savepoint +// while preserving those allocated before it. This is the per- +// savepoint analogue of the full-rollback test above. +#[test] +fn rollback_to_savepoint_truncates_to_savepoint_watermark() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let h1 = tm.allocate(b"before").unwrap(); + tm.savepoint("sp").unwrap(); + let savepoint_watermark = tm.cache.borrow().next_page_id(); + let _h2 = tm.allocate(b"after").unwrap(); + let _h3 = tm.allocate(b"after-2").unwrap(); + assert!(tm.cache.borrow().next_page_id() > savepoint_watermark); + + tm.rollback_to("sp").unwrap(); + assert_eq!( + tm.cache.borrow().next_page_id(), + savepoint_watermark, + "rollback_to must rewind to the savepoint's watermark" + ); + // The pre-savepoint handle must still be readable. + assert_eq!(tm.read(h1).unwrap(), b"before"); + tm.commit().unwrap(); +} + +// An operational error (NoActiveTransaction, DuplicateSavepoint, +// InvalidHandle, etc.) must NOT poison. These are caller mistakes, +// not integrity failures — the manager stays usable. +#[test] +fn operational_error_does_not_poison() { + let mut tm = fresh_manager(); + + // NoActiveTransaction from commit — operational. + assert!(matches!(tm.commit(), Err(ChiselError::NoActiveTransaction))); + assert!(!tm.is_poisoned()); + + // NoActiveTransaction from allocate — operational. + assert!(matches!( + tm.allocate(b"v"), + Err(ChiselError::NoActiveTransaction) + )); + assert!(!tm.is_poisoned()); + + // DuplicateSavepoint — operational. + tm.begin().unwrap(); + tm.savepoint("a").unwrap(); + assert!(matches!( + tm.savepoint("a"), + Err(ChiselError::DuplicateSavepoint(_)) + )); + assert!(!tm.is_poisoned()); + + // InvalidHandle from read — operational. + assert!(matches!(tm.read(999), Err(ChiselError::InvalidHandle(_)))); + assert!(!tm.is_poisoned()); +} + +// Regression test for ISSUES.md I18. Inside commit_inner's +// persist_freemap step, the new-freemap-page allocation must never +// return an id that is still referenced by the currently-committed +// on-disk superblock. The two sources of such at-risk ids are: +// +// (1) `committed_roots.freemap_page` itself — the current +// on-disk freemap page; overwriting it mid-commit destroys +// the committed freemap snapshot. +// (2) Any id in `txn_freed_pages` — pages that held handle +// values reachable through the committed handle table; +// overwriting any of them mid-commit destroys a +// committed value. +// +// A crash in the window between `cache.flush()` and the superblock +// fsync would then leave the last-durable superblock pointing at +// a page whose bytes no longer match what it committed to — +// breaking the core shadow-paging invariant. The fix defers the +// merge of both at-risk sets into `current_freemap` until AFTER +// the new-freemap-page allocate has run, so `FreeMap::allocate_first` +// cannot return any of them during the vulnerable window. +#[test] +fn persist_freemap_does_not_reuse_committed_live_pages() { + let mut tm = fresh_manager(); + + // Commit 1: seed a non-trivial committed state. We need + // persist_freemap to actually materialize a freemap page, + // not take the early-exit path. That requires `txn_freed_pages` + // to be non-empty at commit, which means freeing at least one + // WHOLE data page — R1 slot packing keeps multi-slot data + // pages live even after individual deletes. The simplest way + // to guarantee whole-page frees is to use overflow-sized + // values (> MAX_INLINE_VALUE): each gets its own overflow + // chain, and delete releases every page in the chain into + // txn_freed_pages via Overflow::collect_chain_pages. + let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; + tm.begin().unwrap(); + let h_throwaway = tm.allocate(&big).unwrap(); + let h_live_a = tm.allocate(&big).unwrap(); + let h_live_b = tm.allocate(&big).unwrap(); + let h_live_c = tm.allocate(&big).unwrap(); + tm.delete(h_throwaway).unwrap(); + tm.commit().unwrap(); + + let committed_freemap_page = tm.committed_roots.freemap_page; + assert_ne!( + committed_freemap_page, PAGE_ID_NONE, + "test precondition: commit 1 should have established a freemap page" + ); + + // Commit 2: delete two more handles. release_data_slot pushes + // their data pages into `txn_freed_pages`; those pages are + // still referenced by commit 1's (currently-on-disk) + // superblock at the moment commit_inner runs persist_freemap. + tm.begin().unwrap(); + tm.delete(h_live_a).unwrap(); + tm.delete(h_live_b).unwrap(); + + let frozen_txn_freed: Vec = tm.txn_freed_pages.clone(); + assert!( + !frozen_txn_freed.is_empty(), + "test precondition: deletes should have populated txn_freed_pages" + ); + + tm.commit().unwrap(); + + // The at-risk set: anything that was still live under the + // prior committed superblock at the moment persist_freemap + // started allocating. + let mut still_live_pre_commit = frozen_txn_freed.clone(); + still_live_pre_commit.push(committed_freemap_page); + + let new_freemap_page = tm.committed_roots.freemap_page; + assert!( + !still_live_pre_commit.contains(&new_freemap_page), + "I18: persist_freemap allocated the new freemap page at an id \ + that was still referenced by the last-durable superblock. \ + new_freemap_page={new_freemap_page}, \ + committed_freemap_page was {committed_freemap_page}, \ + txn_freed_pages at commit time = {frozen_txn_freed:?}" + ); + + // Sanity: the un-deleted handle still reads back correctly + // (rules out a subtler corruption that survived the invariant + // check but poisoned the data plane). + assert_eq!(tm.read(h_live_c).unwrap(), big); +} + +// Regression test for ISSUES.md I27. `savepoint_inner` moves +// `txn_freed_pages` into the savepoint record via `std::mem::take`. +// If commit runs with savepoints still on the stack, the pre-fix +// `commit_inner` just called `self.savepoints.clear()` at step 5 +// and those `freed_pages` lists were dropped — never reaching the +// freemap. `persist_freemap` iterates only `self.txn_freed_pages`. +// The post-fix merge in commit_inner flattens every active +// savepoint's `freed_pages` back into `txn_freed_pages` before +// `persist_freemap` runs, so every page freed anywhere in the +// transaction reaches the committed freemap. +// +// Observable via `FreeMap::is_free(&committed_freemap, id)` — the +// freemap's public predicate avoids any reliance on subsequent +// allocator reuse (which depends on `savepoints.is_empty()` too and +// would muddy the test). +#[test] +fn commit_with_active_savepoint_returns_freed_pages_to_freemap() { + let mut tm = fresh_manager(); + + // Seed: enough overflow-sized handles that deleting them + // produces genuine page frees (R1 slot-packing would otherwise + // keep multi-slot data pages live). + let big: Vec = vec![0xCD; MAX_INLINE_VALUE + 32]; + tm.begin().unwrap(); + let h_a = tm.allocate(&big).unwrap(); + let h_b = tm.allocate(&big).unwrap(); + let h_keepalive = tm.allocate(&big).unwrap(); + tm.commit().unwrap(); + + // The leak pattern: delete first, THEN open a savepoint. The + // savepoint captures the accumulated `txn_freed_pages`, leaving + // the outer `txn_freed_pages` empty for the rest of the txn. + tm.begin().unwrap(); + tm.delete(h_a).unwrap(); + tm.delete(h_b).unwrap(); + let frozen_txn_freed: Vec = tm.txn_freed_pages.clone(); + assert!( + !frozen_txn_freed.is_empty(), + "test precondition: overflow-sized deletes should free at least one page" + ); + + tm.savepoint("s").unwrap(); + assert!( + tm.txn_freed_pages.is_empty(), + "savepoint_inner should have moved txn_freed_pages into the savepoint" + ); + + // Commit WITHOUT releasing the savepoint. Pre-fix this silently + // drops savepoint.freed_pages on `savepoints.clear()`; post-fix + // commit_inner merges them into txn_freed_pages first. + tm.commit().unwrap(); + + { + let mut cache = tm.cache.borrow_mut(); + let tree = FreeMapTree::from_roots( + tm.committed_roots.freemap_page, + tm.committed_roots.freemap_depth, + ); + for id in &frozen_txn_freed { + assert!( + tree.is_free(&mut cache, *id).unwrap(), + "I27: freed page {id} should be marked free in the committed \ + freemap tree after commit-with-active-savepoint; \ + frozen_txn_freed={frozen_txn_freed:?}" + ); + } + } + + // Sanity: the surviving handle still reads back (rules out a + // wider corruption that happens to also trip the is_free check). + assert_eq!(tm.read(h_keepalive).unwrap(), big); +} + +// ── Structural-page recycle: adversarial pin-tests ────────────────────── +// +// These three lock down the durability-critical freemap structural recycle +// (docs/specs/2026-06-22 "Structural-page reclamation"): the one-commit +// defer, the rollback reset of the recycle pools, and no lost/double free +// across reuse cycles. They are the GATE for the Phase 2 work — a violation +// is a crash-safety bug, not a cosmetic one. + +// Drive a commit that actually COWs the freemap and supersedes structural +// pages: allocate `n` overflow-sized values (each its own whole page chain), +// commit, then delete `del` of them and commit. The second commit's +// `persist_freemap` marks the freed pages free, COWing the freemap leaf/spine +// and superseding the old freemap pages — exactly the churn the recycle +// model is built around. Returns the surviving handles. +fn structural_churn(tm: &mut TransactionManager, big: &[u8], n: usize, del: usize) -> Vec { + tm.begin().unwrap(); + let mut handles: Vec = (0..n).map(|_| tm.allocate(big).unwrap()).collect(); + tm.commit().unwrap(); + + tm.begin().unwrap(); + for h in handles.drain(..del) { + tm.delete(h).unwrap(); + } + tm.commit().unwrap(); + handles +} + +// PROPERTY 1 — one-commit-defer crash-safety. +// +// A freemap page `P` superseded in transaction `T` is still referenced by +// the pre-`T` superblock until `T` commits, so it may be reused as a +// structural COW target ONLY starting in `T+1` (the one-commit defer). If +// `T+1` ever drew a COW target from a page it superseded THIS transaction, +// a crash before `T+1`'s superblock fsync would corrupt the page the +// recovered (pre-`T+1`) superblock still points at — a durability BUG. +// +// We capture the promoted recycle set at the START of the measured +// transaction `T+1` (== what `begin()` cloned into `structural_reuse`), then +// instrument every structural-reuse pop in `T+1` and assert each popped id is +// drawn from EXACTLY that promoted set — never an id minted or superseded +// within `T+1`. The thread-local reuse log records both pop sites +// (`structural_extend` and `persist_freemap`'s inline closure). +// +// Crucially, `T+1` is a WARMED-UP steady-state transaction doing many +// interleaved allocate+delete ops: each allocate reuses a bitmap-free data +// page, which COWs the freemap leaf in the transaction BODY and supersedes a +// freemap page mid-flight — so a later body allocation in the same +// transaction WOULD pop that just-superseded page if the defer were broken. +// (A single-op transaction supersedes the freemap only at persist_freemap, +// the last structural op, leaving no later pop to expose the bug — this test +// is structured to defeat that blind spot.) +#[test] +fn structural_recycle_one_commit_defer() { + let mut tm = fresh_manager(); + let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; + + // Warm up to steady state: a rotating live population so the bitmap holds + // free data pages (making body allocations COW the freemap) and the + // structural recycle is non-trivially populated. Run several + // delete-then-reallocate commits. + let mut live: Vec = Vec::new(); + tm.begin().unwrap(); + for _ in 0..16 { + live.push(tm.allocate(&big).unwrap()); + } + tm.commit().unwrap(); + for _ in 0..6 { + tm.begin().unwrap(); + let recycled: Vec = live.drain(..8).collect(); + for h in recycled { + tm.delete(h).unwrap(); + } + for _ in 0..8 { + live.push(tm.allocate(&big).unwrap()); + } + tm.commit().unwrap(); + } + + // Capture the promoted recycle the measured transaction inherits, and + // drain the warm-up's reuse log so only the measured transaction is seen. + let promoted: std::collections::HashSet = + tm.pending_structural_frees.iter().copied().collect(); + assert!( + !promoted.is_empty(), + "precondition: warm-up must leave a non-empty deferred recycle" + ); + let _ = take_structural_reuse_log(); + + // T+1 (measured): MANY interleaved allocate+delete ops. Each allocate + // claims a bitmap-free data page (COWing + superseding the freemap in the + // body), each delete frees a page; the heavy interleave means a freemap + // page superseded early in the body has many later body allocations that + // would pop it if the defer leaked same-txn supersedes into the pool. + tm.begin().unwrap(); + for _ in 0..6 { + let recycled: Vec = live.drain(..4).collect(); + for h in recycled { + tm.delete(h).unwrap(); + } + for _ in 0..4 { + live.push(tm.allocate(&big).unwrap()); + } + } + // Pages T+1 superseded so far (the body). persist_freemap adds more at + // commit; both must stay out of the reuse pops. + let superseded_in_t1: std::collections::HashSet = + tm.structural_superseded.iter().copied().collect(); + tm.commit().unwrap(); + + // Read the log IMMEDIATELY after T+1's commit — before any later + // transaction can pop from its OWN (legitimately) promoted recycle and + // pollute the capture with ids that were never inherited here. + let reused_in_t1 = take_structural_reuse_log(); + assert!( + !reused_in_t1.is_empty(), + "precondition: T+1 must actually reuse at least one deferred page \ + (else the defer is untested)" + ); + // NOTE: this block is a weak guard on its own. Under session-COW dedup a + // leaf is COW'd at most once per commit, so a same-transaction supersede + // and its only in-txn reuse pop are the SAME event — they cannot both be + // observed here. The load-bearing defer check is the cross-commit + // REACHABILITY assertion below; this block is kept as a cheap sanity rail. + for id in &reused_in_t1 { + assert!( + promoted.contains(id), + "one-commit-defer VIOLATION: T+1 reused freemap page {id} that was \ + NOT in the promoted recycle set {promoted:?} — it was minted or \ + superseded within T+1, so a pre-commit crash would corrupt the page \ + the last-durable superblock still references" + ); + assert!( + !superseded_in_t1.contains(id), + "one-commit-defer VIOLATION: T+1 reused page {id} that T+1 itself \ + superseded this transaction (still live under the last-durable \ + superblock) — reusing it pre-commit is a crash-safety bug" + ); + } + + // The defer's DURABLE consequence: a page T+1 superseded is now (post- + // commit) dead and queued for T+2 — but it must NOT be reachable in the + // just-committed live tree. A broken defer that re-routed a same-txn + // supersede into the reuse pool would surface here as a pool page still + // live in the committed tree (the corruption a pre-commit crash would + // expose). This is the cross-boundary half of the defer the in-txn pop + // check above cannot see (session-COW dedup COWs each leaf once, so the + // supersede and its only in-txn pop are the same event). + let reachable: std::collections::HashSet = { + let mut cache = tm.cache.borrow_mut(); + let committed = FreeMapTree::from_roots( + tm.committed_roots.freemap_page, + tm.committed_roots.freemap_depth, + ); + committed + .reachable_pages(&mut cache) + .unwrap() + .into_iter() + .collect() + }; + for id in &tm.pending_structural_frees { + assert!( + !reachable.contains(id), + "one-commit-defer VIOLATION: page {id} is queued for reuse in T+2 but is \ + still reachable in the committed freemap tree — a same-transaction \ + supersede leaked into the reuse pool while still live" + ); + } +} + +// PROPERTY 2 — rollback resets the recycle pools and session state. +// +// A rollback must leave the structural recycle exactly as a clean begin would +// see it: `structural_reuse` restored to the committed recycle state (derived +// from `pending_structural_frees`), `structural_superseded` cleared, and +// `freemap_session_owned` cleared. Leaking any of these into the next +// transaction would let it reuse a page the committed tree still references, +// or skip a needed COW on a now-committed page — both corruption. +#[test] +fn structural_recycle_rollback_resets_pools() { + let mut tm = fresh_manager(); + let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; + + // Establish a non-empty committed recycle so the test exercises a real + // restore target, not just emptiness. + let survivors = structural_churn(&mut tm, &big, 8, 4); + let committed_recycle: Vec = tm.pending_structural_frees.clone(); + assert!( + !committed_recycle.is_empty(), + "precondition: a prior commit must leave a non-empty deferred recycle" + ); + + // A transaction that mutates all three pools: allocations + deletes COW + // the freemap (filling session-owned + superseded), and the deletes' frees + // make persist-side reuse pops drain `structural_reuse`. Do NOT commit. + tm.begin().unwrap(); + let _fresh: Vec = (0..6).map(|_| tm.allocate(&big).unwrap()).collect(); + for h in &survivors { + tm.delete(*h).unwrap(); + } + // The session-owned set is populated by freemap COWs on the alloc/delete + // path; assert the test actually dirtied the state it is about to roll + // back (else the reset assertions are vacuous). + assert!( + !tm.freemap_session_owned.is_empty() || !tm.structural_superseded.is_empty(), + "precondition: the pre-rollback churn must have mutated freemap session/supersede state" + ); + + tm.rollback().unwrap(); + + // begin() CLONES `pending_structural_frees` into `structural_reuse`, so an + // aborted transaction's recycle is exactly the pre-transaction one: the + // committed recycle must be intact, and the working pools cleared. + assert_eq!( + tm.pending_structural_frees, committed_recycle, + "rollback must leave the committed deferred recycle intact" + ); + let recycle_after: std::collections::HashSet = + tm.pending_structural_frees.iter().copied().collect(); + let committed_set: std::collections::HashSet = committed_recycle.iter().copied().collect(); + assert_eq!( + recycle_after, committed_set, + "the post-rollback recycle (what the next begin will seed structural_reuse from) \ + must equal the committed recycle state" + ); + assert!( + tm.structural_superseded.is_empty(), + "rollback must clear structural_superseded — those committed-tree pages are \ + still referenced and must never be recycled" + ); + assert!( + tm.freemap_session_owned.is_empty(), + "rollback must clear freemap_session_owned — leaking it would suppress a needed \ + COW and mutate a live committed page in place next transaction" + ); + + // Crucial follow-through: the next transaction must reuse ONLY the + // committed recycle, proving no aborted-transaction page leaked into the + // pool. (An aborted supersede leaking into reuse is a classic double-free.) + let _ = take_structural_reuse_log(); + tm.begin().unwrap(); + let mut next: Vec = (0..8).map(|_| tm.allocate(&big).unwrap()).collect(); + for h in survivors { + tm.delete(h).unwrap(); + } + tm.commit().unwrap(); + for id in take_structural_reuse_log() { + assert!( + committed_set.contains(&id), + "post-rollback transaction reused freemap page {id} not in the committed \ + recycle {committed_set:?} — rollback leaked structural pool state" + ); + } + for h in next.drain(..) { + tm.begin().unwrap(); + tm.delete(h).unwrap(); + tm.commit().unwrap(); + } +} + +// PROPERTY 2b — the orphan sweep must NOT run under a savepoint. +// +// `rollback_to(savepoint)` rewinds the roots + cache watermark but does NOT +// reset the structural recycle streams. The only path that COWs the freemap +// (mutating those streams) while a savepoint is open is the defrag orphan +// sweep. If the sweep ran under a savepoint, it could drain a committed-LIVE +// freemap page into `structural_superseded`; after `rollback_to` (which +// leaves the stream intact) + commit (which promotes it), the NEXT +// transaction would reuse that still-durably-referenced page as a COW target +// and overwrite it — silent durable freemap corruption. +// +// The fix guards `reclaim_freemap_orphans` with `savepoints.is_empty()`. +// This test reproduces the trigger end-to-end and asserts the committed +// freemap tree survives intact. Counterfactual: removing the guard makes the +// committed-tree-intact assertion (or the no-reuse-of-committed-page check) +// fail. +#[test] +fn orphan_sweep_skipped_under_savepoint_preserves_committed_freemap() { + let mut tm = fresh_manager(); + let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; + + // Build a real multi-page freemap with committed structural state so the + // committed tree has actual nodes to corrupt. After this the committed + // freemap root/depth describe a non-trivial tree. + let survivors = structural_churn(&mut tm, &big, 8, 4); + + // Snapshot the committed freemap's free-set and reachable node set BEFORE + // the savepoint episode. These are the ground truth the episode must not + // disturb. + let committed_root = tm.committed_roots.freemap_page; + let committed_depth = tm.committed_roots.freemap_depth; + assert_ne!( + committed_root, PAGE_ID_NONE, + "precondition: a committed freemap tree must exist" + ); + let (free_before, reachable_before): ( + std::collections::BTreeSet, + std::collections::HashSet, + ) = { + let mut cache = tm.cache.borrow_mut(); + let tree = FreeMapTree::from_roots(committed_root, committed_depth); + let reachable: std::collections::HashSet = tree + .reachable_pages(&mut cache) + .unwrap() + .into_iter() + .collect(); + // The set of currently-free ids, scanned over the allocation range. + let total = cache.next_page_id(); + let mut free = std::collections::BTreeSet::new(); + for id in 0..total { + if tree.is_free(&mut cache, id).unwrap() { + free.insert(id); + } + } + (free, reachable) + }; + + // Episode: open a transaction, take a savepoint, forge a freemap orphan, + // and invoke the sweep. With the guard the sweep is a no-op (returns 0) + // and touches NO structural stream; without the guard it would reclaim the + // forged orphan, COWing the committed freemap and draining the superseded + // live page into `structural_superseded`. + tm.begin().unwrap(); + tm.savepoint("sp").unwrap(); + let _orphan = tm.test_forge_freemap_orphan().unwrap(); + let reclaimed = tm.reclaim_freemap_orphans().unwrap(); + assert_eq!( + reclaimed, 0, + "the orphan sweep must be a no-op under an active savepoint (got {reclaimed})" + ); + // The streams the rollback_to does NOT reset must be untouched by the + // sweep, or the rollback leaves dangerous residue. + assert!( + tm.structural_superseded.is_empty(), + "sweep under savepoint leaked into structural_superseded: {:?}", + tm.structural_superseded + ); + + // Roll back to the savepoint (discards the forged page) and commit the + // now-empty transaction. With the guard this commit promotes nothing + // dangerous; without it, the committed-live page the sweep superseded is + // promoted into the reusable pool. + tm.rollback_to("sp").unwrap(); + tm.commit().unwrap(); + + // Next transaction does a freemap-COWing operation (delete a survivor, + // which marks its page free and COWs the freemap). If a committed-live + // freemap page had been promoted into the reuse pool, this is where it + // would be drawn as a COW target and OVERWRITTEN. + tm.begin().unwrap(); + tm.delete(survivors[0]).unwrap(); + tm.commit().unwrap(); + + // The committed freemap tree the ORIGINAL (pre-episode) commit described + // must still be readable and self-consistent: no node it referenced was + // overwritten. We re-open the ORIGINAL committed root/depth and confirm + // its reachable set and free-set are unchanged by everything above. + // (Deleting survivors[0] in the final txn only ADDS a free bit; it never + // removes one and never makes a previously-reachable node unreadable.) + let (free_after, reachable_after): ( + std::collections::BTreeSet, + std::collections::HashSet, + ) = { + let mut cache = tm.cache.borrow_mut(); + let tree = FreeMapTree::from_roots(committed_root, committed_depth); + let reachable: std::collections::HashSet = tree + .reachable_pages(&mut cache) + .unwrap() + .into_iter() + .collect(); + let total = cache.next_page_id(); + let mut free = std::collections::BTreeSet::new(); + for id in 0..total { + if tree.is_free(&mut cache, id).unwrap() { + free.insert(id); + } + } + (free, reachable) + }; + assert_eq!( + reachable_after, reachable_before, + "the original committed freemap tree's node set changed — a committed \ + freemap page was reused-and-overwritten (savepoint guard regression)" + ); + assert_eq!( + free_after, free_before, + "the original committed freemap tree's free-set changed — its on-disk \ + bitmap pages were overwritten by a COW into a reused committed page" + ); +} + +// A corrupt DEAD (non-reachable) page must NOT poison the orphan sweep. +// +// 2026-06-22 review decision ("skip unreadable dead pages"): the sweep scans +// every non-reachable page id to classify it as a freemap orphan. A page that +// is not in the live tree but fails to read because it is GARBAGE cannot be +// confirmed as an orphan, and its corruption is irrelevant (it is dead), so +// the sweep SKIPS it instead of propagating fatal. Contrast: a corrupt page +// REACHABLE from the live tree must still surface fatal via `reachable_pages` +// — that path is deliberately NOT weakened (asserted below). +#[test] +fn corrupt_dead_page_does_not_poison_orphan_sweep() { + let mut tm = fresh_manager(); + let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; + + // Build a committed multi-page freemap and a real orphan to reclaim, so + // the sweep has live work to do AND a corrupt dead page to step over. + let _survivors = structural_churn(&mut tm, &big, 8, 4); + + tm.begin().unwrap(); + let real_orphan = tm.test_forge_freemap_orphan().unwrap(); + let corrupt = tm.test_forge_corrupt_dead_page().unwrap(); + assert_ne!(real_orphan, corrupt); + + // The sweep must succeed (NOT poison) despite the corrupt dead page, and + // must still reclaim the legitimate orphan it can read. + let reclaimed = tm.reclaim_freemap_orphans().unwrap(); + assert!( + reclaimed >= 1, + "sweep must skip the corrupt dead page yet still reclaim the readable \ + orphan (reclaimed={reclaimed})" + ); + assert!( + !tm.is_poisoned(), + "a corrupt DEAD page must not poison the orphan sweep" + ); + tm.commit().unwrap(); + + // Contrast: the live-tree walk MUST still propagate fatal on a corrupt + // node reachable from the committed tree — `reachable_pages` is NOT + // weakened by the dead-page softening above. Corrupt a REAL live node on + // disk (flip its type byte to the wrong PageType but keep the checksum + // valid — a type-corruption reached via a live pointer) and confirm the + // walk surfaces `CorruptPage` for exactly that node. + let live_root = tm.committed_roots.freemap_page; + let live_depth = tm.committed_roots.freemap_depth; + { + let mut cache = tm.cache.borrow_mut(); + // Pick a non-root reachable node so the root's own type check passes + // and the failure happens during descent (the path the softening must + // NOT touch). If the tree is a single root (depth 0), the root IS the + // only node; corrupt it directly. + let tree = FreeMapTree::from_roots(live_root, live_depth); + let reachable: Vec = tree + .reachable_pages(&mut cache) + .unwrap() + .into_iter() + .collect(); + let victim = reachable + .iter() + .copied() + .find(|&id| id != live_root) + .unwrap_or(live_root); + // Wrong type byte, valid checksum: not a bit-flip, a position-type + // corruption that `check_type` rejects. Flip FreeMap<->FreeMapInterior. + { + let buf = cache.get_mut(victim).unwrap(); + let wrong = if buf[0] == crate::page::PageType::FreeMap as u8 { + crate::page::PageType::FreeMapInterior as u8 + } else { + crate::page::PageType::FreeMap as u8 + }; + buf[0] = wrong; + page::stamp_checksum(buf); + } + let err = tree.reachable_pages(&mut cache).unwrap_err(); + assert!( + matches!(err, ChiselError::CorruptPage { .. }), + "reachable_pages must surface fatal CorruptPage on a corrupt LIVE \ + node — the dead-page softening must not weaken the live walk \ + (got {err:?})" + ); + } +} + +// PROPERTY 3 — no lost/double free across reuse cycles. +// +// Churn for several commits, each freeing whole pages. After EACH commit: +// (a) every page freed that commit reads `is_free == true` via a fresh +// `FreeMapTree::from_roots(committed root, depth)` (no lost free); and +// (b) NO page id is simultaneously reachable in the LIVE freemap tree and +// present in `structural_reuse`-derived pool (`pending_structural_frees`) +// — a reuse-pool page MUST be dead (no double-allocation: the same page +// cannot be both a live tree node and a free structural target). +#[test] +fn structural_recycle_no_lost_or_double_free() { + let mut tm = fresh_manager(); + let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; + + // Seed a rotating population so each round both allocates and frees whole + // pages, keeping the freemap COWing every commit. + let mut live: Vec = Vec::new(); + tm.begin().unwrap(); + for _ in 0..10 { + live.push(tm.allocate(&big).unwrap()); + } + tm.commit().unwrap(); + + for round in 0..8u32 { + // Delete half, allocate a fresh half: whole-page frees every commit. + let to_delete: Vec = live.drain(..5).collect(); + tm.begin().unwrap(); + for h in &to_delete { + tm.delete(*h).unwrap(); + } + // Capture this commit's data frees BEFORE commit clears the vector. + let freed_this_commit: Vec = tm.txn_freed_pages.clone(); + for _ in 0..5 { + live.push(tm.allocate(&big).unwrap()); + } + // Re-snapshot: allocations may have reused some freed ids already, + // pulling them back out of the free set. The invariant we pin is on + // the pages STILL freed at commit time, so take the union of frees and + // exclude any id re-claimed as a live value this same transaction. + tm.commit().unwrap(); + + assert!( + !freed_this_commit.is_empty(), + "round {round}: deletes must free at least one whole page" + ); + + // (a) No lost free: every page this commit freed (and did not re-claim + // as a live value) reads free in the committed tree. + let live_set: std::collections::HashSet = live.iter().copied().collect(); + { + let mut cache = tm.cache.borrow_mut(); + let committed = FreeMapTree::from_roots( + tm.committed_roots.freemap_page, + tm.committed_roots.freemap_depth, + ); + for id in &freed_this_commit { + // A freed data page re-claimed as a live value this same + // transaction is correctly NOT free; skip those. + if live_set.contains(id) { + continue; + } + assert!( + committed.is_free(&mut cache, *id).unwrap(), + "round {round}: page {id} was freed this commit but is NOT free in \ + the committed freemap tree — a lost free" + ); + } + } + + // (b) No double-free: a page in the structural reuse pool must be DEAD, + // i.e. never simultaneously reachable in the live freemap tree. Walk + // the committed tree and intersect with the deferred recycle pool. + let reachable: std::collections::HashSet = { + let mut cache = tm.cache.borrow_mut(); + let committed = FreeMapTree::from_roots( + tm.committed_roots.freemap_page, + tm.committed_roots.freemap_depth, + ); + committed + .reachable_pages(&mut cache) + .unwrap() + .into_iter() + .collect() + }; + for id in &tm.pending_structural_frees { + assert!( + !reachable.contains(id), + "round {round}: freemap page {id} is in the structural reuse pool AND \ + still reachable in the live committed freemap tree — a reuse-pool page \ + must be dead (handing it out as a COW target would double-allocate it)" + ); + } + // A reuse-pool page must also not be marked free in the bitmap (the + // two reclamation channels are disjoint by design — a structural page + // rides the in-memory pool, never the bitmap). + { + let mut cache = tm.cache.borrow_mut(); + let committed = FreeMapTree::from_roots( + tm.committed_roots.freemap_page, + tm.committed_roots.freemap_depth, + ); + for id in &tm.pending_structural_frees { + assert!( + !committed.is_free(&mut cache, *id).unwrap(), + "round {round}: structural-reuse page {id} is ALSO marked free in the \ + bitmap — the two reclamation channels overlap, risking a double hand-out" + ); + } + } + } +} + +// The defrag orphan-sweep reclaims a freemap-typed page that a crash would +// have stranded: forge one (a checksum-valid FreeMapInterior unreferenced by +// the live tree and not marked free), sweep, and confirm it now reads free in +// the committed tree. This is the crash-recovery story for the in-memory +// structural recycle — without the sweep these pages leak permanently. +#[test] +fn reclaim_freemap_orphans_marks_lost_freemap_pages_free() { + let mut tm = fresh_manager(); + // Churn so a real multi-page freemap (leaf + spine) exists: overflow-sized + // values give each handle its own page, so deletes free whole pages. + let big: Vec = vec![0xCD; MAX_INLINE_VALUE + 32]; + tm.begin().unwrap(); + let mut hs = Vec::new(); + for _ in 0..40 { + hs.push(tm.allocate(&big).unwrap()); + } + tm.commit().unwrap(); + tm.begin().unwrap(); + for h in hs.iter().step_by(2) { + tm.delete(*h).unwrap(); + } + tm.commit().unwrap(); + + // Forge an orphan exactly as a crash leaves a lost recycle-pool page: + // extended, freemap-typed, unreferenced, not free. + let orphan = tm.test_forge_freemap_orphan().unwrap(); + + tm.begin().unwrap(); + let reclaimed = tm.reclaim_freemap_orphans().unwrap(); + assert!(reclaimed >= 1, "the forged orphan must be reclaimed"); + tm.commit().unwrap(); + + // The orphan now reads free in the committed tree (data-reusable bitmap + // space, disjoint from the structural recycle pool). + let mut cache = tm.cache.borrow_mut(); + let tree = FreeMapTree::from_roots( + tm.committed_roots.freemap_page, + tm.committed_roots.freemap_depth, + ); + assert!( + tree.is_free(&mut cache, orphan).unwrap(), + "reclaimed orphan {orphan} must read free in the committed freemap" + ); +} + +// The sweep's exclusion set is load-bearing: a page CURRENTLY in the live +// in-memory recycle pool (`structural_reuse`) is LIVE recycling state, not an +// orphan. Reclaiming it into the bitmap while it is also pool-reusable would +// double-hand-out the page. Seed the pool with a forged freemap-typed page +// (matching the orphan shape in every respect EXCEPT pool membership) and +// assert the sweep skips it and leaves the pool untouched. +#[test] +fn reclaim_freemap_orphans_excludes_live_recycle_pool() { + let mut tm = fresh_manager(); + // Establish a real freemap tree so the sweep does not early-exit on a + // PAGE_ID_NONE root. + let big: Vec = vec![0xCD; MAX_INLINE_VALUE + 32]; + tm.begin().unwrap(); + let h = tm.allocate(&big).unwrap(); + tm.commit().unwrap(); + tm.begin().unwrap(); + tm.delete(h).unwrap(); + tm.commit().unwrap(); + + // Forge a freemap-typed page that WOULD be flagged as an orphan, then put + // it in the live reuse pool so the exclusion set must spare it. + let pooled = tm.test_forge_freemap_orphan().unwrap(); + + tm.begin().unwrap(); + tm.structural_reuse.push(pooled); + let reclaimed = tm.reclaim_freemap_orphans().unwrap(); + assert_eq!( + reclaimed, 0, + "a page in the live recycle pool must NOT be reclaimed as an orphan" + ); + assert!( + tm.structural_reuse.contains(&pooled), + "the sweep must leave the live recycle pool untouched" + ); + tm.rollback().unwrap(); +} + +// Regression test for ISSUES.md I28. I19 introduced `CacheFull` as +// an **operational** error (documented as "commit or rollback to +// recover"), but `commit_inner` runs `persist_freemap` BEFORE +// `cache.flush()` — and `persist_freemap` itself calls +// `allocate_data_page`, which may trip `maybe_evict`'s ceiling +// check when every existing cache entry is dirty. Pre-fix the +// resulting `CacheFull` propagated out of commit_inner and +// commit()'s poison wrapper poisoned the manager unconditionally. +// The recovery advice ("commit to flush") became impossible to +// follow because commit itself failed. +// +// Post-fix: commit drains the cache BEFORE persist_freemap, so the +// cap is always reachable via eviction when persist_freemap +// itself allocates. CacheFull cannot arise on the commit path. +// +// Setup note: we deliberately use a small `max_pages` so the +// strict cap is cheap to saturate with a few allocations. +// spillway_max_bytes=0 keeps CacheFull reachable (no spillway +// escape hatch), matching the pre-spillway path this test exercises. +#[test] +fn commit_does_not_poison_when_cache_is_at_strict_cap() { + // I66 (ISSUES.md, 2026-05-22): TempDir for RAII cleanup — + // replaces the pre-I66 NamedTempFile + std::mem::forget(file) + // pattern that leaked the temp path on every test run. + let _dir = TempDir::new().unwrap(); + let db_path = _dir.path().join("test.chisel"); + let io = PageIo::open(&db_path, false).unwrap(); + // max_pages=16 — big enough for baseline operations (handle-table + // root + superblocks + freemap) to coexist, small enough that a + // handful of big allocations saturate the strict cap quickly. + // spillway_max_bytes=0 means CacheFull fires at max_pages itself. + let cache = PageCache::new( + io, + 16 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let mut tm = TransactionManager::create_new(cache, 2).unwrap(); + tm.begin().unwrap(); + tm.commit().unwrap(); + + // Seed one handle so the victim transaction can produce a + // non-empty `txn_freed_pages` via delete. Without any frees AND + // with `current_freemap == committed_freemap`, `persist_freemap` + // takes its early-exit path and never allocates — which would + // mean it also cannot trip CacheFull, and the test would fail + // to reproduce the bug. + let big: Vec = vec![0x99; MAX_INLINE_VALUE + 32]; + tm.begin().unwrap(); + let victim = tm.allocate(&big).unwrap(); + tm.commit().unwrap(); + + // Victim transaction: delete to populate txn_freed_pages, then + // allocate until the cache saturates at the strict cap. + // CacheFull from an allocate() is operational — we catch it and + // proceed to commit, which is what we actually want to stress. + tm.begin().unwrap(); + tm.delete(victim).unwrap(); + let mut saturated = false; + for _ in 0..200 { + match tm.allocate(&big) { + Ok(_) => continue, + Err(ChiselError::CacheFull { .. }) => { + saturated = true; + break; + } + Err(e) => panic!("unexpected error during cache-fill setup: {e:?}"), + } + } + assert!( + saturated, + "test precondition: cache did not reach CacheFull in 200 allocations" + ); + assert!( + !tm.is_poisoned(), + "precondition: CacheFull from allocate() is operational and must not poison" + ); + + // The actual I28 check. Pre-fix, `persist_freemap`'s internal + // `allocate_data_page` trips the ceiling and propagates + // CacheFull out of commit_inner; commit()'s poison wrapper + // then sets the poison flag. Post-fix commit drains first. + let result = tm.commit(); + assert!( + result.is_ok(), + "I28: commit over a saturated cache should succeed; got {result:?}" + ); + assert!( + !tm.is_poisoned(), + "I28: CacheFull during commit must not poison — it's operational by design" + ); +} + +// =================================================================== +// BUG#2 (2026-06-16 deepdive): forward/reverse tag-map atomic staging. +// +// `allocate_tagged` maintains two maps that must stay in lockstep: the +// FORWARD map (each chunk's `HandleEntry.tag`, in the handle table) and +// the REVERSE map (tag -> handles, in the membership index, powering +// `handles_with_tag`/delete-by-tag). Before the fix the two roots were +// installed sequentially, so a NON-FATAL `CacheFull`/`SpillwayFull` +// striking between them committed a half-update: +// * allocate: the forward map gained the tag but the reverse did not +// (a tagged chunk invisible to `handles_with_tag`); +// * delete: the tombstone landed but the reverse entry stayed (a +// stale member that later escalates to a FATAL CorruptPage when +// surfaced and acted upon). +// Because those errors are non-fatal they do NOT poison the manager, so +// the half-update survives to `commit()` and onto disk. +// +// Atomic staging computes BOTH candidate roots in a fallible prepare +// phase and installs them together in an infallible phase, so a mid-op +// failure is a no-op for the INSTALLED state (neither map changes, the +// handle id is not burned, inline slot bookkeeping is clean). There is a +// bounded freemap-reuse residue on the abnormal path — see the +// `abort_allocate_prepare` doc and the +// `aborted_tagged_allocate_with_freemap_reuse_is_consistent_and_rollback_reclaims` +// test. The `fail_next_membership_op` hook fires a simulated CacheFull at +// exactly the reverse-map step — the precise divergence window — so these +// are deterministic, not timing-dependent. + +#[test] +fn allocate_membership_failure_leaves_maps_consistent() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + // The id the about-to-fail allocate will (try to) use. + let ghost = tm.current_roots.next_handle; + + tm.fault.fail_next_membership_op.set(true); + let err = tm.allocate_tagged(b"payload", 7).unwrap_err(); + assert!( + matches!(err, ChiselError::CacheFull { .. }), + "expected the injected CacheFull, got {err:?}" + ); + assert!(!tm.is_poisoned(), "a non-fatal CacheFull must not poison"); + + // The forward (tag) and reverse (membership) maps MUST agree. Pre-fix + // the forward map carried tag 7 for `ghost` while the reverse index + // did not — a committed-out-of-sync divergence. + let in_reverse = tm.handles_with_tag(7).unwrap().contains(&ghost); + let in_forward = matches!(tm.tag(ghost), Ok(7)); + assert_eq!( + in_forward, in_reverse, + "forward/reverse tag maps diverged after a failed tagged allocate" + ); + + // Atomic staging makes the failed allocate a no-op for the installed + // state: neither map changed and the handle id was not even burned. + assert!( + !in_forward, + "failed allocate must not install the forward entry" + ); + assert_eq!( + tm.current_roots.next_handle, ghost, + "failed allocate must not burn the handle id" + ); + + // ...and the no-op extends to the R1 packing bookkeeping: the inline + // value's data slot was released, so no phantom live-slot count or ghost + // insert cursor survives to skew later packing / defrag density / page + // reclamation. (Pre-fix this leaked `{page: 1}` and `Some(page)`.) + assert!( + tm.current_live_slots.is_empty(), + "failed allocate left a phantom live-slot count: {:?}", + tm.current_live_slots + ); + assert_eq!( + tm.insert_cursor, None, + "failed allocate left a ghost insert cursor" + ); + + // The manager is still fully usable: a disarmed retry reuses the same + // id and is consistent across BOTH maps, in-session and after commit. + let h = tm.allocate_tagged(b"payload", 7).unwrap(); + assert_eq!(h, ghost, "retry should reuse the un-burned handle id"); + assert_eq!(tm.tag(h).unwrap(), 7); + assert!(tm.handles_with_tag(7).unwrap().contains(&h)); + tm.commit().unwrap(); + assert!(tm.handles_with_tag(7).unwrap().contains(&h)); + // C1: no page reachable from committed_roots may be free in the + // committed freemap — pins that the dropped prepare freed-lists never + // queued a still-referenced page. + assert_no_reachable_page_is_free(&tm); +} + +// The FORWARD-step counterpart: a non-fatal CacheFull during the +// handle-table insert (the step that eagerly bumps the descent depth via +// HandleTable::grow, and on a fresh DB lazily materializes the root). The +// prepare-abort must restore BOTH the depth and the handle-table root +// pointer and release the inline slot — a true no-op. +#[test] +fn allocate_handle_table_failure_leaves_maps_consistent() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + // Fresh DB: the handle table has never been materialized. + let saved_root = tm.current_roots.handle_table_page; + assert_eq!(saved_root, PAGE_ID_NONE, "precondition: empty handle table"); + let ghost = tm.current_roots.next_handle; + + tm.fault.fail_next_handle_table_op.set(true); + let err = tm.allocate_tagged(b"payload", 7).unwrap_err(); + assert!( + matches!(err, ChiselError::CacheFull { .. }), + "expected the injected CacheFull, got {err:?}" + ); + assert!(!tm.is_poisoned(), "a non-fatal CacheFull must not poison"); + + // Complete no-op: the lazily-created root is reverted (back to + // PAGE_ID_NONE), the handle id is not burned, neither map has content, + // and the R1 packing state is clean. + assert_eq!( + tm.current_roots.handle_table_page, saved_root, + "failed forward step left a lazily-materialized handle-table root installed" + ); + assert_eq!(tm.current_roots.next_handle, ghost); + assert!( + matches!(tm.tag(ghost), Err(ChiselError::InvalidHandle(_))), + "ghost handle after failed allocate_tagged must be InvalidHandle" + ); + assert!(tm.handles_with_tag(7).unwrap().is_empty()); + assert!( + tm.current_live_slots.is_empty(), + "failed forward step left a phantom live-slot count: {:?}", + tm.current_live_slots + ); + assert_eq!(tm.insert_cursor, None); + + // Disarmed retry succeeds and is consistent across both maps. + let h = tm.allocate_tagged(b"payload", 7).unwrap(); + assert_eq!(h, ghost, "retry should reuse the un-burned handle id"); + assert_eq!(tm.tag(h).unwrap(), 7); + assert!(tm.handles_with_tag(7).unwrap().contains(&h)); + tm.commit().unwrap(); + assert_no_reachable_page_is_free(&tm); +} + +#[test] +fn delete_membership_failure_leaves_maps_consistent() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let h = tm.allocate_tagged(b"payload", 7).unwrap(); + tm.commit().unwrap(); + + tm.begin().unwrap(); + tm.fault.fail_next_membership_op.set(true); + let err = tm.delete(h).unwrap_err(); + assert!( + matches!(err, ChiselError::CacheFull { .. }), + "expected the injected CacheFull, got {err:?}" + ); + assert!(!tm.is_poisoned(), "a non-fatal CacheFull must not poison"); + + // Pre-fix the tombstone (forward) was installed but the reverse entry + // stayed, so `read(h)` failed while `handles_with_tag` still listed + // `h` — exactly the divergence that later escalates to CorruptPage. + let in_reverse = tm.handles_with_tag(7).unwrap().contains(&h); + let still_live = tm.read(h).is_ok(); + assert_eq!( + still_live, in_reverse, + "forward/reverse tag maps diverged after a failed tagged delete" + ); + // Atomic staging makes the failed delete a COMPLETE no-op. + assert!(still_live, "failed delete must not install the tombstone"); + assert!(in_reverse, "failed delete must not drop the reverse entry"); + + // A disarmed retry succeeds and removes `h` from BOTH maps. + tm.delete(h).unwrap(); + assert!( + matches!(tm.read(h), Err(ChiselError::InvalidHandle(_))), + "handle must be InvalidHandle after successful delete" + ); + assert!(!tm.handles_with_tag(7).unwrap().contains(&h)); + tm.commit().unwrap(); + assert!(!tm.handles_with_tag(7).unwrap().contains(&h)); + assert_no_reachable_page_is_free(&tm); +} + +// Delete's durability guard (delete is the more dangerous direction: a +// stale reverse member for a tombstoned handle later escalates to a FATAL +// CorruptPage). Fail the reverse-map step, COMMIT, then REOPEN: the failed +// delete must have committed NOTHING — `h` is still live AND still a tagged +// member on disk. Pre-fix the tombstone committed while the reverse member +// stayed, so the reopened DB would read `h` as deleted yet still list it. +#[test] +fn delete_membership_failure_survives_reopen_consistently() { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("bug2-del.chisel"); + + let open = |create: bool| -> TransactionManager { + let io = PageIo::open(&path, false).unwrap(); + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + if create { + TransactionManager::create_new(cache, 2).unwrap() + } else { + TransactionManager::open_existing(cache).unwrap() + } + }; + + let h = { + let mut tm = open(true); + tm.begin().unwrap(); + let h = tm.allocate_tagged(b"payload", 9).unwrap(); + tm.commit().unwrap(); + + tm.begin().unwrap(); + tm.fault.fail_next_membership_op.set(true); + assert!(matches!( + tm.delete(h).unwrap_err(), + ChiselError::CacheFull { .. } + )); + // Commit the post-failure state: pre-fix this durably tombstones the + // forward map while the reverse keeps `h`; post-fix it is a no-op. + tm.commit().unwrap(); + h + }; + + // Reopen: the forward (read) and reverse (handles_with_tag) views must + // agree, and since the delete was a no-op both must still see `h`. + let tm = open(false); + let still_live = tm.read(h).is_ok(); + let in_reverse = tm.handles_with_tag(9).unwrap().contains(&h); + assert_eq!( + still_live, in_reverse, + "reopened forward/reverse views diverged for the failed delete's handle" + ); + assert!( + still_live && in_reverse, + "a failed tagged delete must commit nothing: h should survive in both maps \ + (read ok = {still_live}, in reverse index = {in_reverse})" + ); +} + +// Pins the documented `delete_with_tag` error contract: a mid-pass failure +// returns Err (NO TagDropProgress — the dropped-this-pass set is not +// reported), is non-fatal/recoverable, and leaves a CONSISTENT partial +// state because each delete_inner is atomic (BUG#2 staging) — so exactly the +// members processed before the failure are gone from BOTH maps, and the +// partial drop is committable and resumable. +#[test] +fn delete_with_tag_mid_pass_failure_is_consistent_and_drops_progress() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let members = [ + tm.allocate_tagged(b"m0", 5).unwrap(), + tm.allocate_tagged(b"m1", 5).unwrap(), + tm.allocate_tagged(b"m2", 5).unwrap(), + ]; + tm.commit().unwrap(); + + tm.begin().unwrap(); + // Let the 1st delete in the pass commit, fail the 2nd's membership op. + tm.fault.fail_membership_op_after.set(2); + let err = tm.delete_with_tag(5, 3).unwrap_err(); + assert!( + matches!(err, ChiselError::CacheFull { .. }), + "expected the injected CacheFull, got {err:?}" + ); + assert!( + !tm.is_poisoned(), + "a non-fatal mid-pass error must not poison" + ); + + // Exactly one member dropped before the failure (delete_inner is atomic, + // so the failed 2nd delete is a no-op). Don't assume index iteration + // order — count instead. + let live: Vec = members + .into_iter() + .filter(|&h| tm.read(h).is_ok()) + .collect(); + assert_eq!( + live.len(), + 2, + "exactly one member dropped before the failure" + ); + // Forward (read-ok set) and reverse (membership index) agree exactly. + let mut idx = tm.handles_with_tag(5).unwrap(); + let mut live_sorted = live.clone(); + idx.sort_unstable(); + live_sorted.sort_unstable(); + assert_eq!( + live_sorted, idx, + "forward/reverse maps consistent after a failed delete_with_tag pass" + ); + + // The consistent partial drop is committable... + tm.commit().unwrap(); + assert_eq!(tm.handles_with_tag(5).unwrap().len(), 2); + assert_no_reachable_page_is_free(&tm); + + // ...and the bounded loop finishes cleanly on a disarmed retry. + tm.begin().unwrap(); + let (_, complete) = tm.delete_with_tag(5, 3).unwrap(); + assert!(complete, "retry must drain the tag"); + tm.commit().unwrap(); + assert!(tm.handles_with_tag(5).unwrap().is_empty()); + assert_no_reachable_page_is_free(&tm); +} + +// CRITICAL durable-corruption regression (surfaced by the BUG#2 adversarial +// review): `update_inner` must not free the OLD value's pages before the new +// entry is durably installed. A non-fatal CacheFull at the new-value-write +// step — which pre-fix runs AFTER the old free — left the committed handle +// still pointing at pages already queued for reclamation, so commit freed a +// reachable page and a later reuse silently corrupted the live value. +#[test] +fn update_value_write_failure_does_not_free_old_value_pages() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + // Overflow-sized old value, so the whole chain is at stake. + let big = vec![0xABu8; MAX_INLINE_VALUE * 3]; + let h = tm.allocate(&big).unwrap(); + tm.commit().unwrap(); + assert_eq!(tm.read(h).unwrap(), big, "precondition: old value readable"); + + tm.begin().unwrap(); + tm.fault.fail_next_update_value_write.set(true); + let err = tm.update(h, b"replacement").unwrap_err(); + assert!( + matches!(err, ChiselError::CacheFull { .. }), + "expected the injected CacheFull, got {err:?}" + ); + assert!(!tm.is_poisoned(), "a non-fatal CacheFull must not poison"); + + // The failed update is a no-op in-session: the old value is intact. + assert_eq!( + tm.read(h).unwrap(), + big, + "failed update lost the old value in-session" + ); + + // Commit the post-failure state, then assert C1: no page the committed + // handle still references may be free. Pre-fix the old overflow chain was + // queued into txn_freed_pages and freed here while `h` still points at it. + tm.commit().unwrap(); + assert_no_reachable_page_is_free(&tm); + assert_eq!( + tm.read(h).unwrap(), + big, + "failed update lost the old value after commit" + ); + + // End-to-end: churn allocations to force the freemap to hand out any + // wrongly-freed pages, then confirm the old value survived. Pre-fix the + // reachable-but-free chain pages would be reused and overwritten, turning + // read(h) into a CorruptPage / wrong bytes. + tm.begin().unwrap(); + for i in 0..40u8 { + tm.allocate(&[i; 64]).unwrap(); + } + tm.commit().unwrap(); + assert_eq!( + tm.read(h).unwrap(), + big, + "old value corrupted after its prematurely-freed pages were reused" + ); +} + +// Same guarantee for an INLINE old value (the Live old-release path). When +// the old page held only this value, the pre-fix code released it to the +// freemap before the new write, so a failed update committed a +// reachable-but-free data page just like the overflow case. +#[test] +fn update_value_write_failure_does_not_free_old_inline_page() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let old = b"inline-original-value"; + let h = tm.allocate(old).unwrap(); + tm.commit().unwrap(); + + tm.begin().unwrap(); + tm.fault.fail_next_update_value_write.set(true); + assert!(matches!( + tm.update(h, b"replacement").unwrap_err(), + ChiselError::CacheFull { .. } + )); + assert!(!tm.is_poisoned()); + assert_eq!( + tm.read(h).unwrap(), + old, + "failed update lost the inline value" + ); + + tm.commit().unwrap(); + assert_no_reachable_page_is_free(&tm); + assert_eq!(tm.read(h).unwrap(), old); + + // Force reuse, then confirm the old inline value survived. + tm.begin().unwrap(); + for i in 0..40u8 { + tm.allocate(&[i; 64]).unwrap(); + } + tm.commit().unwrap(); + assert_eq!( + tm.read(h).unwrap(), + old, + "old inline value corrupted after a prematurely-freed page was reused" + ); +} + +// Highest-value coverage for the post-write prepare-unwind contract: the new +// value is ALREADY written when the handle-table install fails. The fix must +// (a) leave the OLD location referenced (never freed) and (b) release the +// just-written NEW inline slot so no phantom live-slot / ghost cursor +// survives. Driven via the shared fail_next_handle_table_op hook, which +// handle_table_insert_candidate honors for both allocate and update. (The +// old-overflow-walk failure exit runs the identical new-inline-slot release, +// so this test covers that contract too.) +#[test] +fn update_handle_table_failure_preserves_old_value_and_releases_new_slot() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let old = b"inline-original-value"; + let h = tm.allocate(old).unwrap(); + tm.commit().unwrap(); + + tm.begin().unwrap(); + tm.fault.fail_next_handle_table_op.set(true); + let err = tm.update(h, b"small-new").unwrap_err(); + assert!( + matches!(err, ChiselError::CacheFull { .. }), + "expected the injected CacheFull, got {err:?}" + ); + assert!(!tm.is_poisoned()); + + // (a) The old value is untouched — the old location was never freed. + assert_eq!(tm.read(h).unwrap(), old, "failed update lost the old value"); + // (b) The new value's inline slot was released: the committed baseline + // had exactly one live slot (for `old`), and the failed update must + // leave precisely that — no phantom count for the abandoned new value. + assert_eq!( + tm.current_live_slots.values().sum::(), + 1, + "failed update left a phantom live-slot: {:?}", + tm.current_live_slots + ); + + // Durability: commit, assert C1, force reuse, re-read the old value. + tm.commit().unwrap(); + assert_no_reachable_page_is_free(&tm); + tm.begin().unwrap(); + for i in 0..40u8 { + tm.allocate(&[i; 64]).unwrap(); + } + tm.commit().unwrap(); + assert_eq!( + tm.read(h).unwrap(), + old, + "old value corrupted after reuse following a failed update" + ); +} + +// The headline durability guard: a failed tagged allocate, then COMMIT, +// then REOPEN from the last durable superblock. Pre-fix this persisted a +// forward-map ghost (a committed `HandleEntry.tag` with no reverse-index +// member); post-fix the failed allocate committed nothing. +#[test] +fn allocate_membership_failure_survives_reopen_consistently() { + let dir = TempDir::new().unwrap(); + let path = dir.path().join("bug2.chisel"); + + let open = |create: bool| -> TransactionManager { + let io = PageIo::open(&path, false).unwrap(); + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + if create { + TransactionManager::create_new(cache, 2).unwrap() + } else { + TransactionManager::open_existing(cache).unwrap() + } + }; + + let ghost = { + let mut tm = open(true); + tm.begin().unwrap(); + tm.commit().unwrap(); + + tm.begin().unwrap(); + let ghost = tm.current_roots.next_handle; + tm.fault.fail_next_membership_op.set(true); + assert!(matches!( + tm.allocate_tagged(b"payload", 9).unwrap_err(), + ChiselError::CacheFull { .. } + )); + // Commit the post-failure state: pre-fix this makes the forward-map + // ghost durable; post-fix it commits a clean no-op. + tm.commit().unwrap(); + ghost + }; + + // Reopen and verify the FORWARD map carries no ghost tag-9 entry that + // the REVERSE map is missing. `handles_with_tag(9)` reads the reverse + // map (empty under both code paths); the discriminating check is the + // forward map via `tag(ghost)`. + let tm = open(false); + let forward = tm.tag(ghost); + assert!( + forward.is_err() || forward.as_ref().unwrap() != &9, + "reopened forward map has a ghost tag-9 entry (tag({ghost}) = {forward:?}) \ + with no matching reverse-index member — the maps committed out of sync" + ); + assert!( + tm.handles_with_tag(9).unwrap().is_empty(), + "tag 9 should have no committed members" + ); +} + +// When savepoints are absent (`reuse = true`), a failed tagged-allocate +// prepare may exercise the freemap REUSE path inside `cow_alloc` — the +// candidate COW clears free bits and advances the freemap tree BEFORE +// learning the membership step will fail. The abort restores the installed +// state (maps, handle id, inline slot) but intentionally leaves the +// freemap-reuse residue (bounded allocated-but-unreferenced pages), relying +// on rollback to discard them. This test seeds a committed freemap with free +// pages (so reuse fires), triggers the injection, and verifies: +// +// 1. The failed call returns the injected CacheFull. +// 2. The installed state is unchanged (both maps, next_handle, depth). +// 3. No C1 violation: every page reachable in the committed state is +// consistent after the aborted allocate. +// 4. rollback() returns the freemap to its committed free-set: a fresh +// `FreeMapTree::from_roots(committed_root, committed_depth)` reports +// the same free bits as before the aborted allocate. +#[test] +fn aborted_tagged_allocate_with_freemap_reuse_is_consistent_and_rollback_reclaims() { + let mut tm = fresh_manager(); + + // --- Phase 1: seed a committed freemap with free DATA pages --- + // + // Allocate several overflow-sized values (each occupies at least one + // whole page beyond the superblock/freemap spine), commit, then delete + // some and commit again. The second commit's persist_freemap marks those + // pages free, so the committed freemap now has reuse bits set. + let big: Vec = vec![0xAB; MAX_INLINE_VALUE + 32]; + let mut live_handles = Vec::new(); + tm.begin().unwrap(); + for _ in 0..6 { + live_handles.push(tm.allocate_tagged(&big, 5).unwrap()); + } + tm.commit().unwrap(); + + // Delete the first 3 to free their overflow pages into the committed + // freemap. Tag 5 still has 3 live members after this commit. + tm.begin().unwrap(); + for h in live_handles.drain(..3) { + tm.delete(h).unwrap(); + } + tm.commit().unwrap(); + assert_no_reachable_page_is_free(&tm); + + // Precondition: the committed freemap must have free bits so that the + // next `cow_alloc` exercises the reuse path (not just extend). + let pre_fm_root = tm.committed_roots.freemap_page; + let pre_fm_depth = tm.committed_roots.freemap_depth; + let pre_next_handle = tm.committed_roots.next_handle; + let pre_ht_root = tm.committed_roots.handle_table_page; + let pre_mi_root = tm.committed_roots.membership_index_page; + assert_ne!( + pre_fm_root, + crate::page::PAGE_ID_NONE, + "precondition: committed freemap must be non-empty (need free bits for reuse)" + ); + + // Snapshot the committed free set so we can compare after rollback. + let committed_free_before: std::collections::BTreeSet = { + let tree = FreeMapTree::from_roots(pre_fm_root, pre_fm_depth); + let mut cache = tm.cache.borrow_mut(); + let total = tm.committed_roots.total_pages; + (0..total) + .filter(|&id| tree.is_free(&mut cache, id).unwrap_or(false)) + .collect() + }; + assert!( + !committed_free_before.is_empty(), + "precondition: at least one free page in committed freemap for reuse path" + ); + + // --- Phase 2: begin a new transaction (no savepoints → reuse enabled) + // and inject the membership failure --- + tm.begin().unwrap(); + + // Savepoints must be empty so reuse is live. + assert!( + tm.savepoints.is_empty(), + "precondition: no savepoints, so freemap reuse is enabled" + ); + + let ghost = tm.current_roots.next_handle; + let saved_ht_depth = tm.handle_table.depth(); + + // Fire the injection: `cow_alloc` inside `handle_table_insert_candidate` + // or `membership_insert_candidate` will draw from the committed freemap + // before the prepare fails. + tm.fault.fail_next_membership_op.set(true); + let err = tm.allocate_tagged(&big, 5).unwrap_err(); + assert!( + matches!(err, ChiselError::CacheFull { .. }), + "expected injected CacheFull, got {err:?}" + ); + assert!(!tm.is_poisoned(), "non-fatal CacheFull must not poison"); + + // --- Assertion 1: installed state is unchanged --- + assert_eq!( + tm.current_roots.next_handle, ghost, + "aborted allocate must not consume the handle id" + ); + assert_eq!( + tm.current_roots.handle_table_page, pre_ht_root, + "aborted allocate must not install a new handle-table root" + ); + assert_eq!( + tm.current_roots.membership_index_page, pre_mi_root, + "aborted allocate must not install a new membership-index root" + ); + assert_eq!( + tm.handle_table.depth(), + saved_ht_depth, + "aborted allocate must restore the eagerly-bumped handle-table depth" + ); + // Neither map has `ghost`. + assert!( + matches!(tm.tag(ghost), Err(ChiselError::InvalidHandle(_))), + "ghost handle must not appear in forward map" + ); + assert!( + !tm.handles_with_tag(5).unwrap().contains(&ghost), + "ghost handle must not appear in reverse (membership) map" + ); + // next_handle was not advanced. + assert_eq!( + tm.current_roots.next_handle, pre_next_handle, + "next_handle must equal the committed baseline (not burned)" + ); + + // --- Assertion 2: rollback reclaims the freemap residue --- + tm.rollback().unwrap(); + + // After rollback, committed_roots is unchanged (rollback restores + // current_roots to committed_roots, which did not change mid-transaction). + assert_eq!(tm.committed_roots.freemap_page, pre_fm_root); + assert_eq!(tm.committed_roots.freemap_depth, pre_fm_depth); + + // The free set as seen through the committed freemap tree must equal + // the pre-allocate snapshot: rollback's discard_all_dirty + truncate + // restored the freemap's bit pattern to the committed state. + let committed_free_after: std::collections::BTreeSet = { + let tree = FreeMapTree::from_roots( + tm.committed_roots.freemap_page, + tm.committed_roots.freemap_depth, + ); + let mut cache = tm.cache.borrow_mut(); + let total = tm.committed_roots.total_pages; + (0..total) + .filter(|&id| tree.is_free(&mut cache, id).unwrap_or(false)) + .collect() + }; + assert_eq!( + committed_free_before, committed_free_after, + "rollback must restore the committed freemap free-set: \ + before={committed_free_before:?} after={committed_free_after:?}" + ); + + // --- Assertion 3: C1 holds after rollback (no page reachable from + // committed_roots is marked free) --- + assert_no_reachable_page_is_free(&tm); + + // --- Assertion 4: the manager is still fully operational --- + // A new transaction can retry the allocation and succeed. + tm.begin().unwrap(); + let h = tm.allocate_tagged(&big, 5).unwrap(); + assert_eq!(h, ghost, "retry must reuse the un-burned handle id"); + assert_eq!(tm.tag(h).unwrap(), 5); + assert!(tm.handles_with_tag(5).unwrap().contains(&h)); + tm.commit().unwrap(); + assert_no_reachable_page_is_free(&tm); +} + +// I29: the open-time format-version gate compares MAJOR only, not +// the full u32. Same-major files (regardless of minor) open cleanly; +// different-major files fail fast with UnsupportedFormatVersion. +// This encodes the "sacred within a major version" promise from the +// README: any file written by an N.x binary is readable by every +// other N.x binary, because minor bumps can only add fields in the +// superblock's reserved region (they never break backward reads). +// +// We exercise both halves in one test because they share setup +// (fresh-DB file → patch slots → reopen). Patching both superblock +// slots in lockstep matters because Superblock::select picks the +// highest-counter valid slot; if only slot 0 were patched, slot 1 +// (with the unmodified version) would win on some commits. +#[test] +fn format_version_gate_is_major_only() { + let file = NamedTempFile::new().unwrap(); + let path = file.path().to_path_buf(); + + // Step 0: create a fresh database so there's something on disk + // to patch. The default superblock_count of 2 means we need to + // patch pages 0 AND 1. + { + let io = PageIo::open(&path, false).unwrap(); + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let _ = TransactionManager::create_new(cache, 2).unwrap(); + // drop() releases the flock so the test can read+write the + // file directly below. + } + + // Helper: patch every superblock slot to the given packed + // format_version and re-stamp the trailing checksum so the + // deserialize path still accepts it as a valid slot. + let patch_all_slots = |version: u32, slot_count: usize| { + let mut bytes = std::fs::read(&path).unwrap(); + for slot in 0..slot_count { + let offset = slot * PAGE_SIZE; + bytes[offset + 4..offset + 8].copy_from_slice(&version.to_le_bytes()); + let page_arr: &mut [u8; PAGE_SIZE] = + (&mut bytes[offset..offset + PAGE_SIZE]).try_into().unwrap(); + page::stamp_checksum(page_arr); + } + std::fs::write(&path, &bytes).unwrap(); + }; + + // Case 1: patch both slots to (FORMAT_MAJOR_VERSION, +42). A minor + // bump within the same major must open cleanly — this is the + // whole point of the packed scheme. Pre-fix (exact-match gate) + // this case rejected with UnsupportedFormatVersion. + let minor_bump = page::pack_format_version( + page::FORMAT_MAJOR_VERSION, + page::FORMAT_MINOR_VERSION.wrapping_add(42), + ); + patch_all_slots(minor_bump, 2); + { + let io = PageIo::open(&path, false).unwrap(); + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let tm = TransactionManager::open_existing(cache); + assert!( + tm.is_ok(), + "same-major / different-minor file should open cleanly; got {:?}", + tm.err() + ); + } + + // Case 2: patch to (FORMAT_MAJOR_VERSION + 1, 0). A major bump + // is a real format break and must be refused regardless of minor. + let major_bump = page::pack_format_version(page::FORMAT_MAJOR_VERSION.wrapping_add(1), 0); + patch_all_slots(major_bump, 2); + { + let io = PageIo::open(&path, false).unwrap(); + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + match TransactionManager::open_existing(cache) { + Err(ChiselError::UnsupportedFormatVersion { .. }) => {} + Err(e) => panic!("expected UnsupportedFormatVersion, got {e:?}"), + Ok(_) => panic!("expected UnsupportedFormatVersion, got Ok"), + } + } +} + +// I29 write-gate: a file whose MINOR exceeds this binary's opens READ-ONLY, +// not rejected — within a MAJOR every layout change is additive, so reads +// are safe, but writing would drop fields this binary can't see. A +// same-or-older minor file opens read-write as normal. +#[test] +fn file_minor_newer_than_binary_is_forced_read_only() { + let file = NamedTempFile::new().unwrap(); + let path = file.path().to_path_buf(); + + // Fresh DB so there are 2 superblock slots (pages 0 and 1) to patch. + { + let io = PageIo::open(&path, false).unwrap(); + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let _ = TransactionManager::create_new(cache, 2).unwrap(); + } + + // Patch every slot to (current MAJOR, MINOR + 1) and re-stamp checksums. + let newer_minor = + page::pack_format_version(page::FORMAT_MAJOR_VERSION, page::FORMAT_MINOR_VERSION + 1); + let mut bytes = std::fs::read(&path).unwrap(); + for slot in 0..2 { + let offset = slot * PAGE_SIZE; + bytes[offset + 4..offset + 8].copy_from_slice(&newer_minor.to_le_bytes()); + let page_arr: &mut [u8; PAGE_SIZE] = + (&mut bytes[offset..offset + PAGE_SIZE]).try_into().unwrap(); + page::stamp_checksum(page_arr); + } + std::fs::write(&path, &bytes).unwrap(); + + // Reopen read-WRITE; the gate must force read-only, so begin() fails. + let io = PageIo::open(&path, false).unwrap(); + let cache = PageCache::new( + io, + 1024 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let mut tm = TransactionManager::open_existing(cache) + .expect("a newer-minor file must still OPEN (reads are additive-safe)"); + assert!( + matches!(tm.begin(), Err(ChiselError::ReadOnlyMode)), + "a newer-minor file must be forced read-only" + ); +} + +#[test] +fn allocate_tagged_then_tag_and_handles_with_tag() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let h = tm.allocate_tagged(b"row", 42).unwrap(); + let u = tm.allocate(b"untagged").unwrap(); + tm.commit().unwrap(); + assert_eq!(tm.tag(h).unwrap(), 42); + assert_eq!(tm.tag(u).unwrap(), 0); + assert_eq!(tm.handles_with_tag(42).unwrap(), vec![h]); + assert_eq!(tm.handles_with_tag(99).unwrap(), Vec::::new()); +} + +#[test] +fn handles_with_tag_accumulates_multiple_handles() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let a = tm.allocate_tagged(b"a", 42).unwrap(); + let b = tm.allocate_tagged(b"b", 42).unwrap(); + let c = tm.allocate_tagged(b"c", 42).unwrap(); + tm.commit().unwrap(); + // The reverse index accumulates members; it must not overwrite. + let mut got = tm.handles_with_tag(42).unwrap(); + got.sort(); + let mut want = vec![a, b, c]; + want.sort(); + assert_eq!(got, want); + assert_eq!(tm.tag(a).unwrap(), 42); + assert_eq!(tm.tag(c).unwrap(), 42); +} + +#[test] +fn allocate_tagged_overflow_value_preserves_tag() { + let mut tm = fresh_manager(); + // A value larger than MAX_INLINE_VALUE takes the overflow path in + // allocate_inner; the tag must be stored on the Overflow HandleEntry, + // readable via tag(), and indexed for handles_with_tag(). + let big = vec![0xABu8; MAX_INLINE_VALUE + 100]; + tm.begin().unwrap(); + let h = tm.allocate_tagged(&big, 77).unwrap(); + tm.commit().unwrap(); + assert_eq!(tm.tag(h).unwrap(), 77); + assert_eq!(tm.handles_with_tag(77).unwrap(), vec![h]); + assert_eq!(tm.read(h).unwrap(), big); +} + +#[test] +fn update_preserves_immutable_tag() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let h = tm.allocate_tagged(b"v1", 42).unwrap(); + tm.commit().unwrap(); + tm.begin().unwrap(); + tm.update(h, b"v2").unwrap(); + tm.commit().unwrap(); + // Tags are immutable: changing the value must NOT change the tag, and the + // membership index must still list the handle under its original tag. + assert_eq!(tm.tag(h).unwrap(), 42); + assert_eq!(tm.handles_with_tag(42).unwrap(), vec![h]); +} + +#[test] +fn delete_removes_tagged_chunk_from_index() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let h = tm.allocate_tagged(b"row", 7).unwrap(); + tm.commit().unwrap(); + assert_eq!(tm.handles_with_tag(7).unwrap(), vec![h]); + tm.begin().unwrap(); + tm.delete(h).unwrap(); + tm.commit().unwrap(); + // The tag's last member is gone -> handles_with_tag is empty. + assert_eq!(tm.handles_with_tag(7).unwrap(), Vec::::new()); +} + +#[test] +fn delete_tagged_rejects_wrong_tag() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let h = tm.allocate_tagged(b"row", 5).unwrap(); + // Wrong tag: error, nothing deleted, index intact. + let err = tm.delete_tagged(h, 6).unwrap_err(); + assert!( + matches!(err, ChiselError::TagMismatch { handle, expected: 6, actual: 5 } if handle == h) + ); + assert_eq!(tm.handles_with_tag(5).unwrap(), vec![h]); + // TagMismatch is operational, NOT fatal: a wrong tag must leave the + // manager usable (guards against a future edit moving it into is_fatal). + assert!(!tm.is_poisoned()); + // Right tag: deletes (and self-maintains the index via delete_inner). + tm.delete_tagged(h, 5).unwrap(); + assert_eq!(tm.handles_with_tag(5).unwrap(), Vec::::new()); + tm.commit().unwrap(); +} + +#[test] +fn delete_one_of_two_tagged_keeps_the_other() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let a = tm.allocate_tagged(b"a", 7).unwrap(); + let b = tm.allocate_tagged(b"b", 7).unwrap(); + tm.commit().unwrap(); + tm.begin().unwrap(); + tm.delete(a).unwrap(); + tm.commit().unwrap(); + // Only `a` is removed from the reverse index; `b` survives under tag 7. + assert_eq!(tm.handles_with_tag(7).unwrap(), vec![b]); + assert_eq!(tm.tag(b).unwrap(), 7); +} + +// ── Migrated 2026-05-22 from tests/transactions.rs (I35 reshape) ── +// +// Exercises TransactionManager::open_existing end-to-end: a value +// written through one TransactionManager survives a drop + reopen +// on the same path. The other tests in tests/transactions.rs use +// only the public Chisel API and stay in tests/. +#[test] +fn reopen_preserves_committed_data() { + let file = NamedTempFile::new().unwrap(); + let path = file.path().to_owned(); + let handle; + { + let io = PageIo::open(&path, false).unwrap(); + let cache = PageCache::new( + io, + 64 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let mut txm = TransactionManager::create_new(cache, 2).unwrap(); + txm.begin().unwrap(); + handle = txm.allocate(b"persistent").unwrap(); + txm.commit().unwrap(); + } + { + let io = PageIo::open(&path, false).unwrap(); + let cache = PageCache::new( + io, + 64 * PAGE_SIZE as u64, + 0, + crate::DrainInsertion::LruTail, + crate::SpillwayLocation::InMemory, + ); + let txm = TransactionManager::open_existing(cache).unwrap(); + let data = txm.read(handle).unwrap(); + assert_eq!(data, b"persistent"); + } +} + +#[test] +fn delete_with_tag_drops_in_bounded_batches() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let mut hs = Vec::new(); + for i in 0..10u64 { + hs.push(tm.allocate_tagged(format!("row{i}").as_bytes(), 3).unwrap()); + } + tm.commit().unwrap(); + tm.begin().unwrap(); + let (d1, c1) = tm.delete_with_tag(3, 4).unwrap(); + assert_eq!(d1.len(), 4); + assert!(!c1); + let (d2, c2) = tm.delete_with_tag(3, 100).unwrap(); + assert_eq!(d2.len(), 6); + assert!(c2); + tm.commit().unwrap(); + assert_eq!(tm.handles_with_tag(3).unwrap(), Vec::::new()); + // The chunks themselves are gone too. + for h in hs { + assert!( + matches!(tm.read(h), Err(ChiselError::InvalidHandle(_))), + "handle {h} must be InvalidHandle after delete_with_tag" + ); + } +} + +#[test] +fn delete_with_tag_exact_max_reports_complete() { + // Boundary: exactly `max` members remain, so the max+1 enumeration + // returns exactly `max` and `complete = max <= max` must be TRUE. This + // is the <= vs < edge: a `<` here would falsely report incomplete and + // cost the caller an extra empty pass. (The other delete_with_tag test + // only exercises len > max and len < max, never len == max.) + let mut tm = fresh_manager(); + tm.begin().unwrap(); + for i in 0..5u64 { + tm.allocate_tagged(format!("r{i}").as_bytes(), 8).unwrap(); + } + tm.commit().unwrap(); + tm.begin().unwrap(); + let (deleted, complete) = tm.delete_with_tag(8, 5).unwrap(); + assert_eq!(deleted.len(), 5); + assert!( + complete, + "deleting exactly all members in one max-sized pass must report complete" + ); + tm.commit().unwrap(); + assert_eq!(tm.handles_with_tag(8).unwrap(), Vec::::new()); +} + +#[test] +fn handle_table_depth_restored_after_rolled_back_grow() { + // I99 regression: a rolled-back handle-table grow must not leave the + // in-memory depth too deep. A leaf holds ENTRIES_PER_LEAF (510) ids + // 0..=509; handle 0 is the reserved "no handle" sentinel, so allocation + // starts at id 1. Allocating ids 1..=509 (509 handles) stays at depth 0, + // and the next handle (id 510, where `id >= cap` triggers the grow) goes + // to depth 1. Roll that back, then a committed handle must still read (it + // returned InvalidHandle before the fix). + let mut tm = fresh_manager(); + tm.begin().unwrap(); + for i in 0..509u64 { + tm.allocate(format!("v{i}").as_bytes()).unwrap(); + } + tm.commit().unwrap(); + let baseline = tm.read(5).unwrap(); + tm.begin().unwrap(); + tm.allocate(b"grow").unwrap(); + tm.rollback().unwrap(); + assert_eq!( + tm.read(5).unwrap(), + baseline, + "committed handle lost after rolled-back grow" + ); + // New inserts still work (depth consistent for the next transaction). + tm.begin().unwrap(); + let h = tm.allocate(b"after").unwrap(); + tm.commit().unwrap(); + assert_eq!(tm.read(h).unwrap(), b"after"); +} + +#[test] +fn handle_table_depth_restored_after_rollback_to_savepoint() { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + // Handle 0 is the reserved "no handle" sentinel, so allocation starts at + // id 1: ids 1..=509 (509 handles) stay at depth 0, and the "grow" below + // (id 510, where `id >= ENTRIES_PER_LEAF` triggers the grow) is the one + // that crosses to depth 1, past the savepoint. Handle h holds "v{h-1}". + for i in 0..509u64 { + tm.allocate(format!("v{i}").as_bytes()).unwrap(); + } + tm.savepoint("sp").unwrap(); + tm.allocate(b"grow").unwrap(); // grows depth 0 -> 1 past the savepoint + tm.rollback_to("sp").unwrap(); + // A handle present at the savepoint must still read within the active txn. + // Handle 5 was the i=4 allocation, so it holds "v4". + assert_eq!(tm.read(5).unwrap(), b"v4"); + tm.commit().unwrap(); + assert_eq!(tm.read(5).unwrap(), b"v4"); +} + +#[test] +fn commit_fsync_failure_poisons_at_each_of_the_three_fsyncs() { + // I112: commit performs THREE fsyncs (pre-drain, data-flush, superblock). + // A real IoError at ANY of them must surface as IoError AND poison the + // manager. The FailFsync countdown targets each in turn. A small inline + // value keeps commit to exactly three fsyncs (no spillway: fresh_manager + // sets spillway_max_bytes=0). + for nth in 0..3u32 { + let mut tm = fresh_manager(); + tm.begin().unwrap(); + tm.allocate(b"v").unwrap(); + tm.cache.borrow().io().arm_fault(Fault::FailFsync(nth)); + let result = tm.commit(); + assert!( + matches!(result, Err(ChiselError::IoError(_))), + "commit fsync #{} failure must surface IoError, got {result:?}", + nth + 1 + ); + assert!(tm.is_poisoned(), "fsync #{} failure must poison", nth + 1); + assert!( + matches!(tm.read(0), Err(ChiselError::Poisoned)), + "a poisoned manager rejects all further ops" + ); + } +} + +#[test] +fn commit_write_failure_poisons() { + // I112: a real write_page IoError during commit must surface and poison. + // Target the value's own data page, which is written during commit flush. + let mut tm = fresh_manager(); + tm.begin().unwrap(); + let h = tm.allocate(b"v").unwrap(); + let pid = tm + .handle_live_page_id(h) + .unwrap() + .expect("allocated value has a live data page"); + tm.cache.borrow().io().arm_fault(Fault::FailWritePage(pid)); + let result = tm.commit(); + assert!( + matches!(result, Err(ChiselError::IoError(_))), + "commit write failure must surface IoError, got {result:?}" + ); + assert!(tm.is_poisoned(), "write failure during commit must poison"); +}