Skip to content

fix(onboard): retain durable cleanup recovery - #8083

Closed
ericksoa wants to merge 7 commits into
hold/podman-stack/pr8083-base-9096a968from
fix/managed-bootstrap-durable-cleanup-recovery
Closed

fix(onboard): retain durable cleanup recovery#8083
ericksoa wants to merge 7 commits into
hold/podman-stack/pr8083-base-9096a968from
fix/managed-bootstrap-durable-cleanup-recovery

Conversation

@ericksoa

@ericksoa ericksoa commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Retain process-orphaned managed-bootstrap rollback authority until the owning sandbox service removes the exact held runtime and Docker proves its absence. Recovery now reports every reconciled or retained transaction through a bounded provider-neutral contract, so one unreadable record cannot hide another and a new create blocks only when the retained authority can own its requested name.

Related Issue

Part of #7744.

Stack Position

  • Base: fix(onboard): preserve durable journal compatibility #8080 at 9096a968f13e0c00fdaaa43e8f63e03993f11277
  • This exact head: a2ae901b6acc249811f16d28292f3652cac97a44
  • Stable parent-relative patch ID: 3c61a91c9e58b62f5137f1144c39a426b35c4169
  • Parent-relative scope: 14 files changed, 906 insertions(+), 117 deletions(-)
  • Activation remains dormant; this slice does not advertise buildless or Podman support.

Changes

  • Add a durable owner-cleanup-required journal phase reached only after image-owned shared state is restored and the exact replacement is absent.
  • Keep the restored original quiescent, retain the journal, and withhold terminal rollback evidence until the owner removes the exact runtime and Docker proves absence; unknown presence remains retryable and fail-closed.
  • Wrap image-owned commit-receipt retirement failures in the typed durable-cleanup error so commit authority survives restart and retry.
  • Enumerate journal identities before loading records, isolate per-record failures, and return bounded, normalized, frozen, identity-ordered { receipts, failures } evidence.
  • Block create before network preparation for same-name or sandbox-identity-unknown failures while warning without blocking for exact unrelated sandboxes.
  • Keep central orchestration independent of provider phases and diagnostic codes; MXC-style contract tests exercise the same recovery and create-gating surface without a Docker or Podman switch.
  • Cover staged, cutover, rollback-authorized, owner-cleanup, and shared-state-committed restart paths; mixed success/failure ordering; exact-presence uncertainty; retry durability; idempotence; and all three supported agents.
  • Document the dormant recovery contract and its owner-cleanup ordering.

The additional provider-neutral report is required because throwing on the first unreadable durable record loses evidence for later transactions, while treating every retained record as a global blocker unnecessarily strands unrelated sandbox names. The adapter, journal, Docker recovery, runtime-provider, and sandbox-create tests protect the bounded contract and keep provider-specific routing out of central orchestration.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: The maintainer-approved recovery architecture and dormant activation boundary are tracked in [Epic] Support native Podman with buildless managed onboarding #7744. Exact-head advisors, CodeRabbit, CodeQL, CI, and protected E2E remain required before this draft becomes mergeable.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: src/lib/onboard/managed-bootstrap/README.md; verified dormant provider-neutral recovery, owner-cleanup ordering and durability, bounded { receipts, failures }, same-name and identity-unknown blocking, and unrelated-sandbox warnings against exact-head source and tests. The append-only ancestry merge changes no parent-relative behavior or documentation claim; exact-head focused tests, CLI build and typecheck, Biome lint/format, patch preservation, and diff checks passed.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • Append-only ancestry refresh: exact head a2ae901b6acc249811f16d28292f3652cac97a44 is signed-DCO and GitHub Verified on exact fix(onboard): preserve durable journal compatibility #8080 base 9096a968f13e0c00fdaaa43e8f63e03993f11277; the parent-relative slice patch is 3c61a91c9e58b62f5137f1144c39a426b35c4169. Focused exact-head qualification passed, and fresh public CI and protected E2E are running.
  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project cli src/lib/onboard/managed-bootstrap src/lib/onboard/sandbox-gpu-create-flow.test.ts src/lib/onboard/runtime-provider/runtime-provider-contract.test.ts --coverage=false: 11 files and 173 tests passed. npm run build:cli, npm run typecheck:cli, npm run lint, Biome format, and git diff --check passed at a2ae901b6acc.
  • Applicable broad gate passed — npm run test:fast completed 15,112 passing tests but the resource-saturated local run also produced 44 unrelated timeout/host-condition failures. Public exact-head CI and protected E2E are required for the authoritative broad result; fresh runs are active at the exact head above.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only) — passed with 0 errors and 2 existing non-fatal Fern warnings.
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Aaron Erickson aerickson@nvidia.com

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
# Conflicts:
#	src/lib/onboard/managed-bootstrap/docker-journal.test.ts
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa ericksoa self-assigned this Aug 2, 2026
@copy-pr-bot

copy-pr-bot Bot commented Aug 2, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 62dbf083-96c3-43a1-bf50-98a77e7a9e3d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@ericksoa ericksoa added area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow area: security Security controls, permissions, secrets, or hardening platform: container Affects Docker, containerd, Podman, or images bug-fix PR fixes a bug or regression labels Aug 2, 2026
@github-code-quality

github-code-quality Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit a2ae901 in the fix/managed-bootstra... branch remains at 96%, unchanged from commit c63e7eb in the hold/podman-stack/pr... branch.

TypeScript / code-coverage/cli

The overall coverage in commit a2ae901 in the fix/managed-bootstra... branch remains at 81%, unchanged from commit c63e7eb in the hold/podman-stack/pr... branch.

Show a code coverage summary of the most impacted files.
File hold/podman-stack/pr... c63e7eb fix/managed-bootstra... a2ae901 +/-
src/lib/onboard...trap/adapter.ts 84% 63% -21%
src/lib/onboard...ndbox-create.ts 91% 82% -9%
src/lib/onboard...-run-attempt.ts 95% 88% -7%
src/lib/state/sandbox.ts 85% 81% -4%
src/lib/onboard...eway-service.ts 86% 82% -4%
src/lib/onboard...cker-journal.ts 86% 83% -3%
src/lib/onboard...-transaction.ts 78% 80% +2%
src/lib/onboard...strap/docker.ts 60% 66% +6%
src/lib/onboard...cker-runtime.ts 0% 52% +52%
src/lib/onboard...ntime-create.ts 0% 100% +100%

Updated August 03, 2026 19:22 UTC

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@github-actions

github-actions Bot commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / low confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: Partial review preserved 0 canonical finding(s) and 2 terminology decision(s) before the advisor stopped.

Model lanes

  • GPT-5.6 Terra (primary): Failed after a partial review · low confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 0 blockers · 0 warnings · 0 suggestions

Second-opinion terminology and E2E selections are advisory. They do not change the primary assessment or E2E / PR Gate.

2 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • justified — owner cleanup at src/lib/onboard/managed-bootstrap/README.md:56: Keep `owner-cleanup-required` for this distinct durable phase and retain its defining condition in explanatory text.
  • established — provider-neutral at src/lib/onboard/managed-bootstrap/adapter.ts:269: Keep the established term; the changed use is consistent with existing provider-neutral contracts.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: onboard-repair, onboard-resume, cloud-onboard

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Append #8080 head 9096a96.

Preserve the durable cleanup and recovery slice byte-for-byte.

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa
ericksoa changed the base branch from fix/managed-bootstrap-durable-journal-compatibility to pull-request/8080 August 2, 2026 21:06
@ericksoa
ericksoa changed the base branch from pull-request/8080 to fix/managed-bootstrap-durable-journal-compatibility August 2, 2026 21:06
@ericksoa
ericksoa changed the base branch from fix/managed-bootstrap-durable-journal-compatibility to pull-request/8080 August 2, 2026 21:07
@ericksoa
ericksoa changed the base branch from pull-request/8080 to fix/managed-bootstrap-durable-journal-compatibility August 2, 2026 21:07
@ericksoa
ericksoa changed the base branch from fix/managed-bootstrap-durable-journal-compatibility to hold/podman-stack/pr8083-base-9096a968 August 3, 2026 19:10
ericksoa added a commit that referenced this pull request Aug 4, 2026
## Summary

Hardens the dormant managed-bootstrap path so create outcomes are
explicit, shared-state rollback remains transaction-owned, and recovery
receipts replay durably across runtime providers. This consolidates the
additive source work from #8077, #8078, #8080, and the
already-incorporated behavior from #8083 without registering or
activating a managed runtime.

## Related Issue

Refs #7744

## Changes

- Return terminal managed-bootstrap outcomes and preserve explicit
rollback evidence through Docker sandbox creation.
- Keep application environment and shared-state rollback authority
inside the managed-startup transaction, including environment-neutral
status and rollback probes.
- Move receipt comparison into the provider-neutral adapter, validate
pre-journal snapshot identity, and retain exact atomic leftovers for
durable replay.
- Add focused lifecycle, transaction, compatibility, replay, and
source-shape coverage while keeping the candidate provider inert.
- Preserve the donor heads under
`backup/podman-stack/pr8077-source-83e7fe53`,
`backup/podman-stack/pr8078-source-9d4dc59c`,
`backup/podman-stack/pr8080-source-9096a968`, and
`backup/podman-stack/pr8083-source-a2ae901b`.

The adapter contract is currently required by managed-bootstrap journal
and runtime construction consumers. A Docker-local change is
insufficient because later Podman and MXC-style providers must compare
the same durable receipts without central runtime switches. The
managed-bootstrap adapter, runtime, journal, shared-state, and
source-shape tests protect that boundary.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [ ] Docs updated for user-facing behavior changes
- [x] Docs not applicable — justification: No user-visible provider is
registered or activated in this additive slice; the internal
managed-bootstrap README documents the architecture change.
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: Maintainer-authored
implementation scope under #7744; the provider remains inert and
repository advisors must still clear the exact head before merge.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: Updated `src/lib/onboard/managed-bootstrap/README.md`. The
managed-bootstrap provider remains unregistered and unsupported in
production, so no user-facing `docs/` change is required.
- Agent: Codex Desktop
<!-- docs-review-head-sha: b3973ce -->
<!-- docs-review-agents-blob-sha: 3dd7c24 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification:
Targeted Vitest run covering all 12 changed test files passed 175/175
tests at `b3973cebb50d`.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Local `npm test` was
attempted and encountered widespread unrelated five-second timeouts
across existing installer, package-contract, rebuild, inference, and
policy tests; authoritative sharded CI is pending.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only)
- [ ] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Commit and rollback failures now surface reliably instead of being
silently ignored.
* Improved recovery when runtime finalization or supervisor reconnection
fails.
  * Prevented mismatched container images from advancing through setup.
  * Strengthened rollback protection after a commit becomes durable.

* **Compatibility**
* Added support for valid legacy transaction manifests while rejecting
malformed or incomplete data.

* **Reliability**
* Repeated commit or rollback requests now produce consistent results,
including after acknowledgement failures.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
@ericksoa

ericksoa commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by clean consolidated replacement #8225, now merged. The donor head remains preserved under backup/podman-stack/pr8083-source-a2ae901b.

@ericksoa ericksoa closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: onboarding Onboarding FSM, provider setup, sandbox launch, or first-run flow area: security Security controls, permissions, secrets, or hardening bug-fix PR fixes a bug or regression platform: container Affects Docker, containerd, Podman, or images

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant