perf(e2e): accelerate Hermes rebuild validation

[apurvvkumaria]

Summary

The Hermes rebuild lanes now bootstrap their dependencies once, pull a digest-pinned historical fixture, and carry the exact validated current-base identity through the immutable rebuild handoff. Repeated ubuntu-latest samples reduced the normal median from 1000.5s to 618.5s (38.2%) and the stale-base median from 1068s to 614.5s (42.5%) without weakening state, readiness, replacement, or provenance assertions.

This is incremental progress toward #7144. It does not implement or waive the issue's reusable cross-job BuildKit cache requirement, so #7144 remains open.

Related Issue

Part of #7144

Changes

• Pull the historical Hermes fixture by immutable digest, verify its provenance before use, reuse the prepared current base across rebuild phases, and keep the stale-base refresh boundary intact.
• Separate OpenShell/bootstrap setup from the timed rebuild contract and emit per-phase timing plus runner-resource evidence for comparable repeated samples.
• Preserve state more defensively by failing closed on SQLite backup errors or missing state files and by cleaning temporary immutable base-image handoffs.
• Carry validated base resolution metadata through recreate, managed Dockerfile preparation, and the final image labels so the rebuilt sandbox proves the exact phase-one base identity.
• Reconcile the rebuild handoff with fix(images): require trusted base image overrides #6884's approved trust contract: locally built bases carry the signed per-operation provenance lease, pinned official bases retain the stricter immutable identity/ABI/runtime binder, and arbitrary local aliases remain rejected. Focused base-image and rebuild-preflight tests protect both paths.
• Verify backup/restore, sandbox replacement, readiness, state schema, base layers, image labels, and current/stale lane behavior in focused and live E2E coverage.
• Keep the existing Docker-engine cache security boundary. This PR does not add an external Buildx builder, registry cache, or reusable cross-job BuildKit cache; that requirement remains open in perf(e2e): reduce rebuild-hermes runtime with pinned fixtures and reusable build cache #7144 pending its security-design decision.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: this changes internal rebuild implementation and E2E coverage without adding a command, option, environment variable, migration, or user choice; the required docs review found existing rebuild and base-image validation documentation already accurate.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: provenance remains fail-closed for arbitrary aliases and is adopted only after exact immutable identity, platform, official digest, ABI, and Hermes runtime proof. Focused negative tests cover every rejected mismatch; earlier live samples exercised the normal and stale-base behaviors, and final exact-head E2E remains pending after branch convergence.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: the focused reconciliation suite passed 129 tests; npm run build:cli and npm run typecheck:cli passed; exact-head GitHub CI is running and protected E2E remains pending.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: not run; diff-scoped hooks and focused runtime/support tests cover the changed rebuild contract.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• Bug Fixes
  • Improved Hermes rebuilds by preserving and validating base-image identity/provenance across phases, including correct reuse handling and immutable temporary handoffs.
  • Added safer temporary base-image handoff cleanup with best-effort disposal and retry on cleanup failures.
  • Rebuilds now abort when required sandbox state files fail to back up, unless --force is enabled.
  • Improved SQLite backup/restore reliability via isolated Python execution and stricter empty-output failure handling.
• New Features
  • Hermes rebuild flow now carries pre-resolved base-image provenance into target/image preflights.
  • Added trusted local override support and scoped base-image reuse tagging/cleanup.
• Tests
  • Expanded Hermes end-to-end coverage (base identity evidence, reuse planning, stale-base classification, cleanup/disposal, and workflow boundary validation), including OpenShell binary install/version checks.
• Documentation
  • Clarified --force behavior for manifest-declared state file backup failures and updated sandbox base-image override constraints.

[apurvvkumaria]

@senthilr-nv @cjagwani The requested #7213 follow-up is now pushed on exact head 2c7ce22f5a4cc62096afb06960550e0d90209f48.

• The outer preflight now retains the already-verified canonical imageName@sha256 reference instead of retagging it to an unleased temporary local alias.
• The recreate handoff is scoped to the operation and accepted only when its immutable ref, repository, digest, and resolution metadata agree; mismatch/reuse paths remain fail-closed.
• The protected Hermes test now defaults NEMOCLAW_CLI_BIN to the exact checked-out bin/nemoclaw.js, covering the trusted-base-workflow transition where the job-level variable is absent.
• Added outer-preflight -> recreate override -> real resolver coverage and mismatch/rejection coverage.
• 67 focused tests, 9 E2E-support tests, Biome, repository hooks, and CLI type-check pass. All three new commits are GitHub Verified and DCO is green.

Current-head CI and the protected E2E gate have restarted. Please re-review once the selected Hermes lanes finish.

[senthilr-nv]

The requested changes look good to me. I will approve once the branch is refreshed onto current main and the E2E gate passes.

Stale against current head 26f3b0c: the PR is explicitly scoped as incremental progress toward #7144, reconciles the #6884 trust contract, is refreshed from main, and the exact-head 11-target E2E run completed with the PR-specific rebuild-hermes-stale-base target passing. The remaining aggregate failures are unrelated Hermes transport/gateway lanes.