feat(warp-core): project WAL recovery evidence

[flyingrobots]

Summary

• add WalRecoveryProjection posture and typed obstruction records for recovered WAL projection
• project recovered WAL scans from explicit manifest, writer-epoch, segment seal, locator, and recovery certificate evidence
• add wal_projection_from_recovery coverage for deterministic projection, missing-manifest obstruction, missing-locator obstruction, and non-mutating filesystem projection

Closes #560

Validation

• cargo test -p warp-core wal_projection_from_recovery
• cargo test -p warp-core --test causal_wal_tests
• cargo clippy -p warp-core --lib --test causal_wal_tests -- -D warnings
• cargo check -p warp-core
• cargo fmt --check
• git diff --check
• npx markdownlint-cli2 CHANGELOG.md
• pre-commit hook: cargo clippy -p warp-core --lib, cargo check -p warp-core, markdownlint
• pre-push hook: cargo fmt --all -- --check, cargo check -p warp-core --quiet, cargo test -p warp-core --test causal_wal_tests, prettier touched markdown

Summary by CodeRabbit

• New Features

  • Added recovery projection support for WAL history, producing graph-ready recovery records from available evidence.
  • Recovery now captures more details from manifests, segments, writer epochs, and recovery certificates.

• Bug Fixes

  • Missing manifests or unavailable segment locators now surface typed obstruction states instead of appearing as successful empty results.
  • Improved validation for recovery consistency, including tail state, segment coverage, digest matching, and certificate details.

• Tests

  • Added coverage for deterministic recovery projection and multiple obstruction scenarios.

[coderabbitai]

Warning

Review limit reached

@flyingrobots, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 38 minutes and 48 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1f1dde6b-6242-478c-88b9-c1891192c3cb

📥 Commits

Reviewing files that changed from the base of the PR and between dcb8ab6 and 8205b20.

📒 Files selected for processing (1)

• crates/warp-core/tests/causal_wal_tests.rs

📝 Walkthrough

Walkthrough

Adds WAL recovery projection to warp-core: a pure project_wal_recovery function validates recovered WAL evidence (manifest, segment seals, writer epochs, recovery certificate) and produces typed WalRecoveryProjection records with Absent, Present, or Obstructed postures. A filesystem adapter and a comprehensive integration test covering all obstruction variants are included.

Changes

WAL Recovery Projection

Layer / File(s)	Summary
Recovery certificate digest and RecoveryCertificateRef constructor crates/warp-core/src/causal_wal.rs	Adds WAL_RECOVERY_CERTIFICATE_DOMAIN constant, recovery_certificate_digest domain-separated hash helper, and RecoveryCertificateRef::from_certificate constructor that copies scanned fields and computes the stable digest.
Projection data model crates/warp-core/src/causal_wal.rs	Defines WalRecoverySegmentEvidence, WalRecoveryProjectionPosture (Absent/Present/Obstructed), WalRecoveryProjectionObstruction (all typed variants), and WalRecoveryProjection with absent/present/obstructed construction helpers.
project_wal_recovery pure function crates/warp-core/src/causal_wal.rs	Validates manifest-vs-scan fields, required writer epochs, transaction-to-segment mapping, segment seal coverage, digest correctness, locator availability, mixed-epoch detection, and optional recovery-certificate scan mismatch; returns WalRoot on success or typed obstructions on failure.
Filesystem adapter and segment evidence scanner crates/warp-core/src/causal_wal.rs	filesystem_wal_recovery_segment_evidence scans segment files, rejects torn tails, computes digests and seal posture, and attaches relative/absolute storage locators. project_filesystem_wal_recovery feeds this evidence into project_wal_recovery.
Integration test and changelog crates/warp-core/tests/causal_wal_tests.rs, CHANGELOG.md	wal_projection_from_recovery builds and seals a filesystem WAL, asserts deterministic projection across runs, drives all obstruction variants with mutated inputs, and confirms filesystem immutability post-recovery. Changelog updated under Unreleased.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

Frames recovered, epochs aligned,
Each segment sealed, each digest signed.
Missing manifest? Obstruction returned —
No silent empty where certainty's earned.
The WAL projects, deterministic, clean. 🔒

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: projecting WAL recovery evidence in warp-core.
Linked Issues check	✅ Passed	The PR implements recovery-based projection records, typed obstruction states, deterministic recovery projection, and non-mutating behavior required by #560.
Out of Scope Changes check	✅ Passed	The changes stay within WAL recovery projection and test coverage, with only a changelog update outside code logic.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cycle/560-projection-from-wal-recovery

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f4df9a800d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[flyingrobots]

Code Lawyer resolution update:

Source	Issue	Resolution	Commit
Self-review	Empty recovery projection with supplied evidence could return Absent instead of surfacing missing manifest evidence.	project_wal_recovery now only returns Absent for evidence-free empty scans with no manifest and a clean tail.	dbc64967, completed by dcb8ab6a
Codex PRRT_kwDOQH8Wr86MwBBb	Preserve tail obstructions for empty recovery scans.	Non-clean empty recovery reports now flow through TailPostureObstructed and MissingManifest, with witness assertions.	dcb8ab6a
Codex PRRT_kwDOQH8Wr86MwBBe	Reject seals that do not cover recovered commits.	Segment evidence now obstructs when a seal is missing or below the recovered segment last LSN.	dcb8ab6a
Codex PRRT_kwDOQH8Wr86MwBBg	Validate recovery certificates before embedding them.	Certificate first/last LSN, replay count, and tail posture are compared to the recovery report before projection.	dcb8ab6a
Codex PRRT_kwDOQH8Wr86MwBBh	Reject duplicate segment evidence instead of overwriting it.	Duplicate segment ids now produce DuplicateSegmentEvidence before root projection.	dcb8ab6a
Codex PRRT_kwDOQH8Wr86MwBBj	Verify segment digests against recovered frames.	Projection recomputes each segment digest from recovered frames and obstructs on mismatch.	dcb8ab6a
Codex PRRT_kwDOQH8Wr86MwBBl	Preserve manifest mismatch obstructions.	Filesystem projection now maps concrete manifest validation mismatches to typed projection obstructions.	dcb8ab6a

Validation:

• cargo fmt
• cargo test -p warp-core wal_projection_from_recovery
• cargo test -p warp-core --test causal_wal_tests
• cargo clippy -p warp-core --lib --test causal_wal_tests -- -D warnings
• cargo check -p warp-core
• cargo fmt --check
• git diff --check
• npx markdownlint-cli2 CHANGELOG.md
• pre-commit warp-core hook
• pre-push causal_wal_tests lane

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/warp-core/tests/causal_wal_tests.rs`:
- Around line 377-418: Add test coverage for the clean no-evidence Absent path
in project_wal_recovery. The current assertions only exercise empty inputs that
still end up Obstructed; add one case using RecoveryScanReport with no
transactions and RecoveryTailPosture::Clean plus no segment evidence/manifests,
and assert the projection posture is WalRecoveryProjectionPosture::Absent. Keep
the new assertion near the existing empty-input cases in causal_wal_tests.rs so
regressions in the fast path are caught.
- Around line 510-532: Add the missing test coverage in causal_wal_tests for the
filesystem manifest-segment-count mismatch by exercising
project_filesystem_wal_recovery with a manifest whose segment count differs from
what the adapter expects, and assert that
WalRecoveryProjectionObstruction::ManifestSegmentCountMismatch is reported. Use
the existing filesystem_manifest_mismatch test block as the locator, and make
sure the assertion checks the expected and actual values in the correct order so
it would catch the swapped-argument bug in project_filesystem_wal_recovery.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 50914cc5-bddf-47ce-a27e-19e9fce4b545

📥 Commits

Reviewing files that changed from the base of the PR and between 078e4f7 and dcb8ab6.

📒 Files selected for processing (3)

• CHANGELOG.md
• crates/warp-core/src/causal_wal.rs
• crates/warp-core/tests/causal_wal_tests.rs

[flyingrobots]

Code Lawyer follow-up update:

Source	Issue	Resolution	Commit
CodeRabbit PRRT_kwDOQH8Wr86MwIkX	Add coverage for the clean/no-evidence Absent posture.	wal_projection_from_recovery now asserts all-empty clean scans return Absent, no root, and no obstructions.	8205b20d
CodeRabbit PRRT_kwDOQH8Wr86MwIkZ	Add filesystem manifest segment-count mismatch coverage.	The filesystem adapter path now has a witness for ManifestSegmentCountMismatch { expected: 2, actual: 1 }, preserving the projection obstruction contract.	8205b20d

Validation:

• cargo fmt
• cargo test -p warp-core wal_projection_from_recovery
• cargo test -p warp-core --test causal_wal_tests
• cargo clippy -p warp-core --lib --test causal_wal_tests -- -D warnings
• cargo check -p warp-core
• cargo fmt --check
• git diff --check
• npx markdownlint-cli2 CHANGELOG.md
• pre-commit warp-core hook
• pre-push causal_wal_tests lane