Skip to content

fix: improve gateway lifecycle recovery#953

Merged
kjw3 merged 9 commits intomainfrom
fix/gateway-lifecycle-recovery-final
Mar 26, 2026
Merged

fix: improve gateway lifecycle recovery#953
kjw3 merged 9 commits intomainfrom
fix/gateway-lifecycle-recovery-final

Conversation

@kjw3
Copy link
Contributor

@kjw3 kjw3 commented Mar 26, 2026

Supersedes #952 and the closed #908. This branch rebuilds the same work as a single verified signed commit so it can satisfy the branch signature requirements.

Summary

  • preserve a healthy shared nemoclaw gateway across repeat onboarding
  • reconcile live OpenShell sandbox state during connect and status instead of trusting stale local registry entries
  • classify restart/rebuild lifecycle failures so users get deterministic guidance instead of generic transport errors
  • extend double-onboard coverage so creating a second sandbox does not break the first

Issues

  • addresses #849
  • addresses part of #859
  • improves #888 by distinguishing:
    • gateway trust rotation / handshake failure
    • gateway metadata exists but the restarted API still refuses connections
    • gateway rebuilt and the old sandbox no longer exists

Security

  • no secret persistence added
  • no TLS downgrade or bypass added
  • no destructive auto-recovery of healthy sandboxes
  • no new shell-out paths beyond resolved openshell wrappers

Validation

npx vitest run test/cli.test.js test/onboard.test.js test/onboard-readiness.test.js test/registry.test.js
npx eslint bin/nemoclaw.js bin/lib/onboard.js test/cli.test.js test/onboard.test.js
npx tsc -p jsconfig.json --noEmit
bash -n test/e2e/test-double-onboard.sh
shellcheck test/e2e/test-double-onboard.sh

Brev CPU Validation

Environment:

  • brev-cpu
  • instance: kj-nemoclaw-cpu-20260325-155447
  • branch commit tested: 43cf8eb

Validated on a real disposable Linux host:

  • onboard sandbox A / onboard sandbox B path preserves the shared gateway and keeps the first sandbox reachable
  • after openshell gateway stop + openshell gateway start --name nemoclaw, NemoClaw now surfaces a precise post-restart classification instead of a generic transport failure
  • after destructive gateway rebuild, NemoClaw removes the stale local sandbox entry when the old sandbox is gone
  • rerunning onboard after that rebuild recreates the sandbox cleanly and returns to Ready

Residual

  • This PR does not make OpenShell gateway restarts durable. It makes the failure modes explicit, safer, and easier to recover from.
  • On the tested Brev CPU host, I did not find a safe non-destructive recovery once the restarted gateway API entered the persistent Connection refused state. That remains a gateway/runtime limitation, not something this PR tries to bypass.

Summary by CodeRabbit

  • New Features

    • Reuse healthy gateways automatically, verify live sandbox/gateway state before actions, and add a non-fatal gateway startup mode for recovery.
  • Bug Fixes

    • Improved gateway health detection, smarter port/forward handling, enhanced stale registry reconciliation, and consolidated dashboard/control UI routing and URL/token behavior.
  • Tests

    • Expanded unit, integration, and e2e tests for lifecycle recovery, ANSI-safe outputs, logs forwarding, and registry reconciliation.
  • Chores

    • Clean staged build context (remove Python venv/cache) and add helper cleanup script.
  • Documentation

    • Minor macOS first-run checklist formatting and simplified NVIDIA Endpoints label.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 26, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Detects and reuses a healthy shared NemoClaw OpenShell gateway, adds live sandbox reconciliation for connect/status flows, introduces recovery-aware gateway startup modes, runs staged build cleanup for blueprint artifacts, and expands unit/CLI/e2e tests covering gateway lifecycle and registry reconciliation.

Changes

Cohort / File(s) Summary
Core gateway lifecycle & onboard helpers
bin/lib/onboard.js
Added ANSI-tolerant parsing (stripAnsi, getActiveGatewayName) and isGatewayHealthy(). Refactored startup into startGatewayWithOptions(..., { exitOnFailure }), added startGatewayForRecovery(), updated preflight() to reuse healthy gateways (skip 8080 check) and destroy stale state only when detected. Exported new functions.
CLI — openshell invocation, sandbox reconciliation & recovery
bin/nemoclaw.js
Replaced generic runner usage with getOpenshellBinary, runOpenshell, captureOpenshell. Introduced sandbox/gateway reconciliation helpers (getSandboxGatewayState, getNamedGatewayLifecycleState, getReconciledSandboxGatewayState, ensureLiveSandboxOrExit) and recoverNamedGatewayRuntime. Made sandbox commands async and recovery-aware; updated connect/status/logs/destroy flows.
Staged build cleanup & setup
scripts/clean-staged-tree.sh, scripts/setup.sh
Added clean-staged-tree.sh to remove Python artifacts (.venv, .pytest_cache, __pycache__) and invoked it from scripts/setup.sh for the nemoclaw-blueprint build context.
Tests — unit, CLI, gateway-cleanup, e2e
test/cli.test.js, test/onboard.test.js, test/gateway-cleanup.test.js, test/e2e/test-double-onboard.sh, test/onboard-readiness.test.js
Expanded CLI test harness (runWithEnv) and many integration tests for logs forwarding, ANSI handling, registry mutation/reconciliation, recovery flows, and lifecycle messaging. Added isGatewayHealthy unit tests and mocked startGateway() integration test. Updated gateway-cleanup test expectations and rewrote double-onboard e2e into lifecycle-recovery flow with fake OpenAI endpoint and timeout handling.
Misc docs
README.md
Small formatting tweak in macOS first-run checklist (inserted an empty bullet).

Sequence Diagram(s)

sequenceDiagram
    actor User
    participant CLI as CLI (bin/nemoclaw.js)
    participant Onboard as Onboard (bin/lib/onboard.js)
    participant OpenShell as OpenShell
    participant Registry as Local Registry

    rect rgba(200,150,100,0.5)
    Note over User,Registry: Initial onboarding may recreate gateway
    User->>CLI: onboard
    CLI->>Onboard: startGateway()
    Onboard->>OpenShell: openshell status / gateway info
    Onboard->>OpenShell: destroy/start gateway (if stale)
    OpenShell-->>Onboard: gateway ready
    Onboard-->>CLI: success (new gateway)
    end

    rect rgba(100,150,200,0.5)
    Note over User,Registry: Reuse healthy shared gateway
    User->>CLI: onboard
    CLI->>Onboard: startGateway()
    Onboard->>OpenShell: openshell status / gateway info
    OpenShell-->>Onboard: Connected & named
    Onboard->>OpenShell: select existing gateway
    Onboard-->>CLI: success (reused gateway)
    end

    rect rgba(150,200,100,0.5)
    Note over User,Registry: Sandbox connect/status reconciliation
    User->>CLI: sandbox connect / status
    CLI->>Onboard: ensureLiveSandboxOrExit
    Onboard->>OpenShell: openshell sandbox get / logs
    OpenShell-->>Onboard: live sandbox state
    Onboard->>Registry: compare with local registry
    alt sandbox live
        Onboard->>OpenShell: forward/connect
    else registry stale
        Onboard->>Registry: remove stale entry
        Onboard-->>User: guidance / recovery messaging
    end
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🐰
I sniffed the gateways, found one strong,
Reused the path and hopped along.
Two sandboxes, both intact,
No stale traces left to track,
Onboard sings a lively song!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 19.05% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'fix: improve gateway lifecycle recovery' directly aligns with the main objective of the PR, which is to improve how the system handles gateway lifecycle recovery and reuse.
Linked Issues check ✅ Passed The PR comprehensively addresses all coding objectives from issue #908: preserves healthy gateways via isGatewayHealthy() and startGatewayWithOptions(), reconciles live sandbox state via getSandboxGatewayState() and ensureLiveSandboxOrExit(), classifies failure modes in sandboxStatus(), and extends test coverage for gateway reuse and recovery flows.
Out of Scope Changes check ✅ Passed All changes are in-scope: gateway health detection, lifecycle classification, recovery flows, async sandbox operations, test coverage expansion, and supporting infrastructure like clean-staged-tree.sh align with stated objectives.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/gateway-lifecycle-recovery-final

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link

🚀 Docs preview ready!

https://NVIDIA.github.io/NemoClaw/pr-preview/pr-953/

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
bin/lib/onboard.js (1)

1487-1488: Centralize this staged-tree scrubber.

The same .venv / .pytest_cache / __pycache__ cleanup now lives here and in scripts/setup.sh, so the next artifact tweak has to land in two places. A shared helper/script would keep the legacy setup path aligned with onboard.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@bin/lib/onboard.js` around lines 1487 - 1488, Duplicate cleanup commands for
removing .venv, .pytest_cache and __pycache__ are present in onboard.js
(run(...) calls) and scripts/setup.sh; extract them into a single centralized
"staged-tree scrubber" (either a small shell script like
scripts/clean-staged-tree.sh or a JS helper exported from
bin/lib/cleanStagedTree.js) and replace the inline run(...) invocations in
onboard.js and the matching snippet in scripts/setup.sh to call that central
helper. Ensure the central helper performs the same operations (rm -rf for .venv
and .pytest_cache and find ... -name __pycache__ -prune -exec rm -rf {} +) and
that onboard.js's call uses the existing run(...) wrapper semantics (preserve {
ignoreError: true } behavior) or that scripts/setup.sh invokes the script with
identical error-tolerant behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@bin/nemoclaw.js`:
- Around line 100-120: The connected check in getNamedGatewayLifecycleState uses
/Connected/i which also matches "Disconnected"; update the connected detection
to match the exact status token/line in cleanStatus (e.g. replace the
/Connected/i.test(cleanStatus) usage with a more specific regex such as one that
anchors to the status line like /^Status:\s*Connected\b/im.test(cleanStatus) or
a whole-word match for "Connected" so "Disconnected" does not match), leaving
the rest of the function logic unchanged and still using the cleanStatus
variable.

In `@test/e2e/test-double-onboard.sh`:
- Around line 411-418: The current acceptance regex for gateway_status_output
incorrectly treats "Removed stale local registry entry" as a valid gateway-stop
outcome; modify the grep pattern in the gateway status check to remove that
phrase so only lifecycle/recovery messages ("Recovered NemoClaw gateway
runtime", "gateway is no longer configured after restart/rebuild", "gateway is
still refusing connections after restart", "gateway trust material rotated after
restart") count as passes, and keep the existing pass/fail behavior
(pass(...)/fail(...)) around gateway_status_output. After this check, add an
explicit assertion that the registry still contains SANDBOX_B by calling the
registry query used elsewhere in the test (the same registry listing helper that
references SANDBOX_B) and fail the test if SANDBOX_B is missing; reference
gateway_status_output, SANDBOX_B, and the pass/fail helpers when implementing
these changes.

---

Nitpick comments:
In `@bin/lib/onboard.js`:
- Around line 1487-1488: Duplicate cleanup commands for removing .venv,
.pytest_cache and __pycache__ are present in onboard.js (run(...) calls) and
scripts/setup.sh; extract them into a single centralized "staged-tree scrubber"
(either a small shell script like scripts/clean-staged-tree.sh or a JS helper
exported from bin/lib/cleanStagedTree.js) and replace the inline run(...)
invocations in onboard.js and the matching snippet in scripts/setup.sh to call
that central helper. Ensure the central helper performs the same operations (rm
-rf for .venv and .pytest_cache and find ... -name __pycache__ -prune -exec rm
-rf {} +) and that onboard.js's call uses the existing run(...) wrapper
semantics (preserve { ignoreError: true } behavior) or that scripts/setup.sh
invokes the script with identical error-tolerant behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 17d70415-acbc-418a-b174-fa1a4f01e638

📥 Commits

Reviewing files that changed from the base of the PR and between 0acb2e4 and 70fabaa.

📒 Files selected for processing (7)
  • bin/lib/onboard.js
  • bin/nemoclaw.js
  • scripts/setup.sh
  • test/cli.test.js
  • test/e2e/test-double-onboard.sh
  • test/gateway-cleanup.test.js
  • test/onboard.test.js

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
bin/lib/onboard.js (1)

188-203: Consider using regex for ANSI stripping.

The manual character-by-character approach works but is harder to maintain and has O(n²) complexity due to string concatenation. The existing isSandboxReady function at line 171 already uses a regex for ANSI stripping.

♻️ Suggested simplification using regex
 function stripAnsi(value = "") {
-  let cleaned = "";
-  for (let i = 0; i < value.length; i += 1) {
-    if (value.charCodeAt(i) === 27 && value[i + 1] === "[") {
-      i += 2;
-      while (i < value.length && /[0-9;]/.test(value[i])) {
-        i += 1;
-      }
-      if (value[i] === "m") {
-        continue;
-      }
-    }
-    cleaned += value[i] || "";
-  }
-  return cleaned;
+  // eslint-disable-next-line no-control-regex
+  return value.replace(/\x1b\[[0-9;]*m/g, "");
 }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@bin/lib/onboard.js` around lines 188 - 203, The stripAnsi function currently
iterates character-by-character and builds a string which is harder to maintain
and can be O(n²); replace its implementation with a single regex-based removal
(same approach used in isSandboxReady) to strip ANSI escape sequences and return
value.replace(ANSI_REGEX, '') where ANSI_REGEX matches CSI/escape sequences;
update stripAnsi to use that regex and ensure it defaults value to an empty
string as before.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@bin/lib/onboard.js`:
- Around line 188-203: The stripAnsi function currently iterates
character-by-character and builds a string which is harder to maintain and can
be O(n²); replace its implementation with a single regex-based removal (same
approach used in isSandboxReady) to strip ANSI escape sequences and return
value.replace(ANSI_REGEX, '') where ANSI_REGEX matches CSI/escape sequences;
update stripAnsi to use that regex and ensure it defaults value to an empty
string as before.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0b3e9fc7-003d-423c-b21f-81102d9fc253

📥 Commits

Reviewing files that changed from the base of the PR and between e560a17 and fdb20b9.

📒 Files selected for processing (1)
  • bin/lib/onboard.js

@kjw3 kjw3 merged commit 6ae809a into main Mar 26, 2026
8 checks passed
@kjw3 kjw3 deleted the fix/gateway-lifecycle-recovery-final branch March 26, 2026 03:23
TSavo pushed a commit to wopr-network/nemoclaw that referenced this pull request Mar 26, 2026
* fix: improve gateway lifecycle recovery (NVIDIA#953)

* fix: improve gateway lifecycle recovery

* docs: fix readme markdown list spacing

* fix: tighten gateway lifecycle review follow-ups

* fix: simplify tokenized control ui output

* fix: restore chat route in control ui urls

* refactor: simplify ansi stripping in onboard

* fix: shorten control ui url output

* fix: move control ui below cli next steps

* fix: swap hard/soft ulimit settings in start script (NVIDIA#951)

Fixes NVIDIA#949

Co-authored-by: KJ <kejones@nvidia.com>

---------

Co-authored-by: KJ <kejones@nvidia.com>
Co-authored-by: Emily Wilkins <80470879+epwilkins@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants