Skip to content

fix(rebuild): settle expired Shields before success - #8713

Merged
apurvvkumaria merged 2 commits into
mainfrom
codex/fix-8697-hermes-shields-deadline-main
Aug 10, 2026
Merged

fix(rebuild): settle expired Shields before success#8713
apurvvkumaria merged 2 commits into
mainfrom
codex/fix-8697-hermes-shields-deadline-main

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

A long Hermes rebuild can outlive an inherited Shields auto-restore timer. NemoClaw now settles an expired deferred timer through the existing fenced Shields-up path before reporting rebuild success, which prevents lockdown from racing the first post-rebuild tool call.

The live E2E lane now records bounded, redacted HTTP, runtime, fixture, Shields, and cleanup evidence for this boundary.

Related Issue

Fixes #8697

Changes

  • Restrict the production change to rebuild windows that defer automatic Shields restoration while the lifecycle owner is alive. If that timer expires during rebuild, the existing Shields-up operation settles the timer before rebuild returns success.
  • Preserve ordinary backup behavior when the sandbox already had Shields down.
  • Record gateway process identity before rebuild and after MCP restoration.
  • Record post-rebuild fixture requests without credentials or request bodies. The record includes only the authentication result, method, path, response status, and result presence.
  • Capture bounded Hermes logs, filesystem permissions, managed-supervisor status, gateway identity, and Shields status. Existing artifact redaction removes known credentials.
  • Lower Shields before MCP and sandbox cleanup. The cleanup registry uses last-in, first-out order, so evidence capture runs first, Shields-down runs second, and older cleanup operations run afterward.

Confirmed evidence

Root cause and correction

The failure was not stale MCP configuration or credential rotation. The overdue Shields timer made /sandbox/.hermes read-only immediately after rebuild released its lifecycle lock. Rebuild now settles that timer with the existing fenced Shields-up state transition before it reports success.

Remaining validation

  • Run the canonical trusted mcp-bridge (hermes) lane on the exact PR commit.
  • Confirm the first post-rebuild request returns MCP_AUTH_REWRITE_OK::nemoclaw-authenticated-mcp-proof.
  • Confirm the fixture records exactly one authenticated tools/call with the rotated provider credential.
  • Confirm the final evidence records the expected gateway identities, Shields state, redaction, and cleanup result.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: The change corrects an internal timer-settlement race and adds test evidence. It does not change a command, option, configuration, output contract, or recovery procedure.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Independent Codex security review covered all nine repository categories and found no actionable finding. The trusted-main live E2E run remains pending.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: Existing rebuild documentation already states that the detached auto-lock timer remains authoritative until rebuild commits a successful Shields-up state. docs/ is unchanged.
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: 8 timer tests, 63 rebuild/reconciliation tests, 88 Hermes transaction/convergence/integrity tests, 13 E2E support tests, and 2 bounded HTTP diagnostic tests passed. npm run typecheck:cli and npx prek run --from-ref origin/main --to-ref HEAD passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not applicable. The diff changes one lifecycle decision and focused live evidence.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • Bug Fixes

    • Fixed backup protection windows that could remain unlocked after their automatic restore timer expired.
    • Rebuilding protection settings now correctly settles expired timers and reapplies lockdown when required.
  • Tests

    • Expanded coverage for timer expiration, relocking behavior, and rebuild workflows.
    • Improved diagnostics captured during runtime and gateway lifecycle testing, with sensitive values redacted.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria self-assigned this Aug 10, 2026
@copy-pr-bot

copy-pr-bot Bot commented Aug 10, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The pull request updates backup Shields relocking for expired deferred timers. It also adds redacted Hermes runtime diagnostics and gateway identity capture to the rebuild E2E flow, while removing brittle generated-script assertions.

Changes

Shields relocking

Layer / File(s) Summary
Timer-aware relocking
src/lib/actions/sandbox/backup-shields-window.ts, test/rebuild-shields-window.test.ts
Relocking now proceeds when deferred restoration has expired. Tests cover timer settlement with and without calling shieldsUp.

Hermes rebuild diagnostics

Layer / File(s) Summary
Runtime evidence collection
test/e2e/live/mcp-bridge-hermes-http.ts, test/e2e/live/mcp-bridge.test.ts, test/e2e/support/mcp-bridge-hermes-http.test.ts
The rebuild flow records gateway identity, post-rebuild MCP results, Shields status, and redacted runtime diagnostics. Generated-script assertions now omit specific curl details.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: bug-fix, area: security

Suggested reviewers: cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Linked Issues check ❓ Inconclusive The changes address timer settlement, diagnostics, cleanup, and targeted coverage for #8697, but trusted-main E2E acceptance remains pending. Run the trusted-main mcp-bridge (hermes) E2E lane on OpenShell 0.0.101 and confirm the post-rebuild proof and fixture-call criteria.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the primary change: settling expired Shields before rebuild success.
Out of Scope Changes check ✅ Passed The timer fix and E2E diagnostic, cleanup, and test changes directly support the objectives in #8697.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-8697-hermes-shields-deadline-main

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit d5d0788 in the codex/fix-8697-herme... branch remains at 96%, unchanged from commit 3ac3a77 in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit d5d0788 in the codex/fix-8697-herme... branch remains at 82%, unchanged from commit 3ac3a77 in the main branch.

Show a code coverage summary of the most impacted files.
File main 3ac3a77 codex/fix-8697-herme... d5d0788 +/-
src/lib/actions...ields-window.ts 100% 100% 0%
src/lib/sandbox...rce-identity.ts 86% 86% 0%
src/lib/tunnel/services.ts 80% 80% 0%
src/lib/agent/g...tart-scripts.ts 97% 100% +3%

Updated August 10, 2026 08:45 UTC

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 3 warnings · 1 suggestion
  • Model comparison: normalized findings differ; normalized terminology decisions differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 3 more warnings, 1 more suggestion.
1 terminology difference from the second opinion

Advisory only. These are normalized differences from the primary terminology receipt.

  • deferred timer at src/lib/actions/sandbox/backup-shields-window.ts:63: selected only by the second-opinion lane as define.
2 additional E2E selections from the second opinion

Advisory only. The primary lane did not select these E2E jobs or targets.

  • rebuild-hermes: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
  • rebuild-hermes-stale-base: The completed second-opinion lane identified E2E coverage that the primary lane omitted.

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

4 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • established — Shields timer at test/rebuild-shields-window.test.ts:161: Keep `Shields timer` for the automatic-restoration timer. Use `elapsed` only when the deadline state matters.
  • established — gateway identity at test/e2e/live/mcp-bridge-hermes-http.ts:59: Keep `gateway identity` for the lifecycle-bound process identity.
  • established — managed supervisor at test/e2e/live/mcp-bridge-hermes-http.ts:57: Keep `managed supervisor` for the lifecycle process.
  • justified — runtime diagnostics at test/e2e/live/mcp-bridge.test.ts:1201: Keep `runtime diagnostics` for the bounded post-rebuild artifact.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite against this exact revision.

Recommended E2E: cloud-inference, cloud-onboard, security-posture, mcp-bridge, mcp-bridge-dev, onboard-repair, onboard-resume, rebuild-openclaw, state-backup-restore

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@apurvvkumaria
apurvvkumaria marked this pull request as ready for review August 10, 2026 08:46

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/live/mcp-bridge-hermes-http.ts`:
- Around line 47-61: Update buildHermesMcpRuntimeDiagnosticsScript and its
FAILURE_BODY_EMITTER flow to avoid emitting raw service logs, process output,
command lines, or other secret-bearing content. Capture only explicitly
allowlisted, bounded diagnostic fields, and ensure every emitted value is fully
redacted before artifact capture rather than relying solely on API_SERVER_KEY
replacement or the existing four caller-supplied redactions.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6572d05e-565a-4410-9119-4fbffdaa8409

📥 Commits

Reviewing files that changed from the base of the PR and between 3ac3a77 and d5d0788.

📒 Files selected for processing (5)
  • src/lib/actions/sandbox/backup-shields-window.ts
  • test/e2e/live/mcp-bridge-hermes-http.ts
  • test/e2e/live/mcp-bridge.test.ts
  • test/e2e/support/mcp-bridge-hermes-http.test.ts
  • test/rebuild-shields-window.test.ts
💤 Files with no reviewable changes (1)
  • test/e2e/support/mcp-bridge-hermes-http.test.ts

Comment on lines +47 to +61
export function buildHermesMcpRuntimeDiagnosticsScript(): string {
return [
"set -eu",
"set -a",
"[ ! -f /sandbox/.hermes/.env ] || . /sandbox/.hermes/.env",
"set +a",
"{",
'for log in /tmp/nemoclaw-start.log /tmp/gateway.log; do printf \'== %s ==\\n\' "$log"; tail -n 100 "$log" 2>&1 || true; done',
"printf '%s\\n' '== permissions =='",
"stat -c '%a %U:%G %n' /sandbox /sandbox/.hermes /sandbox/.hermes/logs 2>&1 || true",
"printf '%s\\n' '== managed supervisor =='",
"cat /run/nemoclaw/gateway-control/status 2>&1 || true",
"printf '%s\\n' '== gateway identity =='",
"cat /sandbox/.hermes/runtime/gateway.pid 2>&1 || true",
`} | /usr/bin/python3 -I -S -c ${shellQuote(FAILURE_BODY_EMITTER)} /dev/stdin`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Complete redaction before emitting runtime diagnostics.

Lines 54-61 include raw service log output. The local emitter replaces only API_SERVER_KEY. The caller also supplies only four named redaction values. Other credentials, tokens, request bodies, or command lines in these logs can reach the artifact unredacted.

Emit allowlisted diagnostic fields, or redact all secret-bearing values before artifact capture. As per path instructions, capture only bounded, redacted artifacts and do not emit credentials, tokens, request bodies, command lines, or child-process output to logs.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/live/mcp-bridge-hermes-http.ts` around lines 47 - 61, Update
buildHermesMcpRuntimeDiagnosticsScript and its FAILURE_BODY_EMITTER flow to
avoid emitting raw service logs, process output, command lines, or other
secret-bearing content. Capture only explicitly allowlisted, bounded diagnostic
fields, and ensure every emitted value is fully redacted before artifact capture
rather than relying solely on API_SERVER_KEY replacement or the existing four
caller-supplied redactions.

Source: Path instructions

@apurvvkumaria
apurvvkumaria enabled auto-merge (squash) August 10, 2026 08:49

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@apurvvkumaria
apurvvkumaria merged commit f4b1e01 into main Aug 10, 2026
77 checks passed
@apurvvkumaria
apurvvkumaria deleted the codex/fix-8697-hermes-shields-deadline-main branch August 10, 2026 08:58
@github-actions github-actions Bot added the v0.0.106 Release target label Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v0.0.106 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hermes MCP rebuild reports verified on OpenShell v0.0.101 but first provider-backed tool call returns HTTP 500

2 participants