feat(runtime): persist engine lifecycle recovery

[ericksoa]

Summary

Adds a dormant, provider-neutral lifecycle ledger that binds snapshot, clone, rebuild, backup, restore, and recovery mutations to the exact persisted engine authority from the preceding slice. It records restart-safe phases, one exclusive process-owned mutation lease, target-position-bound commands, and completion receipts without selecting a provider, activating Podman, or interpreting provider-owned acceleration state.

Related Issue

Part of #7744.

Changes

• Define versioned lifecycle actions and resource roles for snapshot creation, snapshot clone, rebuild, backup, restore, and recovery.
• Bind each transaction to the exact provider, engine authority, endpoint authority, source and target runtime IDs, runtime-state digest, and acceleration-state digest.
• Record prepared, mutation-authorized, and completed phases before publishing an exact retirement tombstone that prevents transaction-ID reuse.
• Revalidate the persisted authority, injected engine, runtime handles, state digest, and exact execution lease before and after target-bound commands and during restart recovery.
• Admit only one live executor per transaction; recover a dead process owner's durable lease before another callback may enter.
• Require each provider command to declare the exact runtime target position, rejecting IDs smuggled into a non-target option.
• Add a private append-only store with canonical parsing and fail-closed symlink, ownership, permission, conflict, and partial-state handling.
• Add MXC-style tests for every lifecycle action, concurrent executor exclusion, dead-owner restart recovery, drift, command-target scope, exact cleanup, idempotency, and source-shape inventory.
• Document the dormant lifecycle ownership and deferred GPU/provider activation boundary in src/lib/onboard/managed-bootstrap/README.md.

The lifecycle ledger is required by #7744 so later provider slices can recover mutation authority without process-global engine state. A direct orchestration switch would bypass the pluggable provider boundary. persisted-engine-lifecycle.test.ts protects the contract.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: This slice is internal and dormant; its ownership and activation boundary is recorded in the internal managed-bootstrap README.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Maintainer-directed inert slice in [Epic] Support native Podman with buildless managed onboarding #7744; exact authority, restart, drift, filesystem, command-scope, and tombstone boundaries have focused adversarial tests.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: Independently reviewed src/lib/onboard/managed-bootstrap/README.md against WRITING.md and docs/CONTRIBUTING.md. The text accurately documents the provider-neutral ledger, restart lease, exact runtime-target authority, retirement tombstone, and dormant activation boundary. Exact-head CLI build, typecheck, 29/29 focused lifecycle tests, 2/2 source-boundary tests, repository checks, Biome, markdownlint, and git diff --check passed.
• Qualification carry-forward: Exact head baa495a on base a2ae17e preserves the byte-identical previously reviewed slice diff (stable patch ID 0cecddf85f35e7d35d2692590e281b9bb63b071b; binary diff SHA-256 3d8e40c4ac4c60aae97500ddd0a9565f7287cc0ee4f85c952fff3b31182f8451), and its source and documentation tree is unchanged from reviewed head 52ecd0b.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• Qualification-only exact head/base: baa495a / a2ae17e; signed-DCO append-only cascade, stable-patch proof, byte-identical slice diff, final-tree equality, and normal pre-push hooks passed.
• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: npm run build:cli and npm run typecheck:cli passed; npx vitest run --project cli src/lib/onboard/runtime-provider/persisted-engine-authority.test.ts src/lib/onboard/runtime-provider/persisted-engine-lifecycle.test.ts passed 29/29; npx vitest run --project integration test/runtime-provider-source-shape.test.ts passed 2/2; npm run checks:repository, the test-size and test-conditional scans, Biome, git diff --check, exact-slice commitlint, and the pre-push hooks passed at exact head 52ecd0b76.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result:
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Aaron Erickson aerickson@nvidia.com

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9dd4e3c9-3d3a-454d-bbcf-1eeafa547835

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 9209d31 in the feat/persisted-engin... branch remains at 96%, unchanged from commit a7237ea in the feat/podman-image-bo... branch.

Show a code coverage summary of the most impacted files.

File	feat/podman-image-bo... a7237ea	feat/persisted-engin... 9209d31	+/-
nemoclaw/src/se...ntial-filter.ts	100%	0%	-100%
nemoclaw/src/sh...er-boundary.cts	96%	0%	-96%
nemoclaw/src/se...ot-sanitizer.ts	95%	0%	-95%
nemoclaw/src/co...test-fixture.ts	81%	0%	-81%
nemoclaw/src/co...ration-state.ts	93%	94%	+1%

TypeScript / code-coverage/cli

The overall coverage in commit 9209d31 in the feat/persisted-engin... branch remains at 81%, unchanged from commit e9fb170 in the feat/podman-image-bo... branch.

Show a code coverage summary of the most impacted files.

File	feat/podman-image-bo... e9fb170	feat/persisted-engin... 9209d31	+/-
src/lib/onboard...atcher-lease.ts	0%	75%	+75%
src/lib/onboard...ne-lifecycle.ts	0%	83%	+83%
src/lib/onboard...an-lifecycle.ts	0%	84%	+84%
src/lib/onboard...trap-journal.ts	0%	86%	+86%
src/lib/onboard...-replacement.ts	0%	87%	+87%
src/lib/onboard...-transaction.ts	0%	88%	+88%
src/lib/onboard...eld-workload.ts	0%	88%	+88%
src/lib/onboard...ne-authority.ts	0%	89%	+89%
src/lib/onboard...al-inference.ts	0%	90%	+90%
src/lib/onboard...an-preflight.ts	0%	96%	+96%

---

_{Updated August 03, 2026 23:56 UTC}

[github-actions]

PR Review Advisor — Informational

Advisor assessment: Informational / low confidence
Next action: Review the warnings below.
Findings: 0 blockers · 1 warning · 0 suggestions
Status: Partial review preserved 1 canonical finding(s) and 0 terminology decision(s) before the advisor stopped.

Model lanes

• GPT-5.6 Terra (primary): Failed after a partial review · low confidence · 0 blockers · 1 warning · 0 suggestions
• Nemotron 3 Ultra (second opinion): Failed

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: inference-routing, network-policy, onboard-repair, onboard-resume, cloud-onboard

1 warning · 0 suggestions

Warnings

Warnings do not block.

PRA-1 Warning — Remove or connect the inert persisted lifecycle framework

• Location: src/lib/onboard/runtime-provider/persisted-engine-lifecycle.ts:1
• Category: architecture
• Problem: The persisted lifecycle framework has no current production consumer. Its durable phases, lease recovery, tombstones, exact-command wrapper, and six operation types therefore add a second lifecycle system that cannot yet enforce a supported runtime contract.
• Impact: Future lifecycle callers can adopt a broad, unexercised persistence and mutation abstraction instead of a contract derived from the active caller, which increases recovery and authority drift risk.
• Recommendation: Delete the inert persisted lifecycle framework from this PR, or connect the smallest required subset to a named current production consumer and add that consumer's contract test.
• Verification: Read production imports of persisted-engine-lifecycle.ts and confirm whether any active lifecycle path calls preparePersistedEngineLifecycle or executePersistedEngineLifecycle.
• Test coverage: If a current lifecycle consumer exists, add a contract test that invokes it through this boundary across restart and authority-change paths; otherwise delete the inert lifecycle module and its isolated tests.
• Evidence: src/lib/onboard/runtime-provider/persisted-engine-lifecycle.ts adds the new durable lifecycle store and executePersistedEngineLifecycle API. src/lib/onboard/managed-bootstrap/README.md states that snapshot, clone, rebuild, backup, restore, and recovery can prepare an inert lifecycle ledger and that later provider slices must supply dependencies. test/runtime-provider-source-shape.test.ts inventories the new module as a provider source but does not identify a production caller.

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

[apurvvkumaria]

Comment — reviewed exact head baa495a. Persisted engine lifecycle recovery is safe to merge and I found no blocking defect. Non-blocking fast follow: add a recovery contract for the case where an external mutation succeeds but the callback or receipt publication fails. The mutation-authorized retry can otherwise repeat the operation unless each provider reconciles or resumes idempotently; a narrow follow-up test would make that requirement explicit.

[apurvvkumaria]

Comment — reviewed exact head 9209d31 against its stacked base. The provider-neutral lifecycle and new host-local inference authority remain dormant; receipts use closed schemas and digest-pinned images, engine authority is operation-scoped, and requested NVIDIA devices must match the qualified CDI inventory. Focused validation passes 53 of 53 tests. No blocking correctness, security, compatibility, or regression defect found. Non-blocking fast follow: before activating this boundary, normalize host device paths before applying the raw /dev/nvidia rejection, or allowlist the exact non-CDI device forms emitted by supported launchers, so dot-segment aliases cannot bypass the intended CDI-only rule. A narrow follow-up PR covering that translator edge is sufficient; it is not reachable from the current trusted launcher subset and does not need to delay this PR. The branch currently conflicts with its stacked base, so any conflict-resolution revision needs re-review.

[cv]

Reviewed commit 9209d3180a in bottom-up stack order. This PR depends on #8056, which is not approval-ready. This lifecycle layer conflicts with its current base and has failing image, audit, CI, and E2E evidence. Review remains blocked until the parent is reconstructed or repaired, this branch is restacked on the accepted parent commit, and current CI, security, and documentation evidence passes for the resulting commit. No approval is granted for the current branch.