Skip to content

perf(policy): batch onboarding preset application#3502

Merged
cv merged 5 commits into
mainfrom
perf/batch-onboard-policy-presets
May 14, 2026
Merged

perf(policy): batch onboarding preset application#3502
cv merged 5 commits into
mainfrom
perf/batch-onboard-policy-presets

Conversation

@cv
Copy link
Copy Markdown
Collaborator

@cv cv commented May 14, 2026

Summary

Batch built-in policy preset application during onboarding so selected presets are merged in memory and submitted to OpenShell with a single policy set --wait. This reduces repeated gateway policy waits while preserving final policy and registry state.

Changes

  • Added applyPresets() in src/lib/policy/index.ts to merge multiple built-in presets, preserve endpoint disclosure logs, submit one policy update, and record applied preset names.
  • Updated onboarding preset reconciliation to batch newly-selected built-in presets when no per-preset access override is active, while keeping removals and custom/non-built-in presets on the existing sequential path.
  • Added tests for batched policy merging, preset-diff stubs, policy tier resume flows, and an installer preflight fixture that now reaches the shared host preflight gate deterministically.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • make docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

Summary by CodeRabbit

  • Improvements
    • Smarter preset sync that removes deselected presets and only applies newly selected ones, with optimized batching for built-in presets.
    • Reliable mutation retry behavior when applying/removing presets to handle transient sandbox states.
  • New Features
    • Multi-preset apply operation to apply several built-in presets in a single step.
  • Tests
    • Added and updated tests covering batch preset application, preset-sync behavior, and mutation retry flows.

Review Change Stack

@cv cv self-assigned this May 14, 2026
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 14, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 13cec6c1-44e5-41ad-8ead-66d54c77a9c2

📥 Commits

Reviewing files that changed from the base of the PR and between 4c07771 and 420f28d.

📒 Files selected for processing (2)
  • src/lib/onboard/policy-preset-sync.ts
  • test/policy-preset-sync.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/lib/onboard/policy-preset-sync.ts

📝 Walkthrough

Walkthrough

Adds a batch preset application API (applyPresets) to merge multiple built-in presets in one policy-set call, extracts preset-sync and retry logic into src/lib/onboard/policy-preset-sync.ts, wires the helper into onboarding, and updates tests/mocks to exercise the batch flow.

Changes

Batch Preset Application

Layer / File(s) Summary
Batch preset application implementation
src/lib/policy/index.ts
applyPresets(sandboxName, presetNames) validates sandbox name, fetches current policy once, merges each preset's network_policies, writes the merged policy to a temp file, applies it in a single policy set call, and updates registry.sandbox.policies.
Preset-sync helper with retry
src/lib/onboard/policy-preset-sync.ts
Adds waitForPolicyMutation (retry wrapper) and syncPresetSelection (computes deselected/new presets, removes deselected, applies built-in presets in batch and others individually, supports optional per-preset access).
Onboarding preset selection integration
src/lib/onboard.ts
Replaces the removed local syncPresetSelection() with the imported implementation from ./onboard/policy-preset-sync.
Integration tests & preset-sync test
test/policies.test.ts, test/policy-preset-sync.test.ts
Adds an integration-style test for applyPresets() and a compiled-script test for syncPresetSelection asserting batching and ordering behavior.
Test suite mock updates
test/onboard-preset-diff.test.ts, test/policy-tiers-onboard.test.ts
Test mocks extended to implement policies.applyPresets() so bulk preset application is recorded across many onboarding/preset-resolution tests.
Docker stub timing adjustment
test/install-preflight.test.ts
Docker probe stub updated to fail only after nemoclaw is linked into NPM_PREFIX, allowing preflight to reach CLI-link before Docker reachability fails.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~40 minutes

Possibly related PRs

  • NVIDIA/NemoClaw#2442: Both PRs modify src/lib/onboard.ts's preset-application flow in syncPresetSelection().

Suggested labels

refactor, enhancement: performance

Suggested reviewers

  • jyaunches
  • prekshivyas
  • ericksoa
  • cjagwani

Poem

🐰 I hop through presets, tidy and spry,

Merging built-ins in one single try.
Tests clap their paws, the registry grins,
I nibble mismatches and stash small wins.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'perf(policy): batch onboarding preset application' directly and clearly describes the main change: batching built-in policy preset application during onboarding to improve performance.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch perf/batch-onboard-policy-presets

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

ESLint skipped: no ESLint configuration detected in root package.json. To enable, add eslint to devDependencies.


Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/lib/onboard.ts (1)

9320-9341: Run the full onboarding E2E matrix before merge for this core-flow change.

This modifies preset reconciliation in the core onboarding path; run the recommended onboarding E2E jobs to catch regressions in sandbox lifecycle and provider/channel flows.

As per coding guidelines: src/lib/onboard.ts “contains core onboarding logic” and changes here affect full sandbox creation/configuration flow, with the listed E2E recommendation set.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard.ts` around lines 9320 - 9341, This change touches core
onboarding preset reconciliation (applyPresets, applyPreset,
waitForPolicyMutation, newlySelected, accessByName in src/lib/onboard.ts); run
the full onboarding E2E matrix (all sandbox
lifecycle/provider/channel/onboarding permutations) before merging to catch
regressions—execute the project's recommended onboarding E2E jobs (including
sandbox creation, preset application, provider/channel flows, and cleanup)
against this branch and report any failures, re-run locally if flaky, and only
merge after green runs and any failures are fixed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard.ts`:
- Around line 9320-9335: syncPresetSelection has grown with batching logic;
extract the split-and-apply orchestration into a new helper in the policy module
and keep syncPresetSelection as a thin delegator. Create a function (e.g.,
orchestratePresetApplications or applyPresetsBatch) in the policy module that
takes sandboxName and newlySelected, computes builtInPresetNames from
policies.listPresets(), splits newlySelected into builtInNewlySelected and
remainingNewlySelected, and calls waitForPolicyMutation + policies.applyPresets
for the batch and waitForPolicyMutation + policies.applyPreset for each
remaining item; then replace the inlined block in syncPresetSelection with a
single call to the new policy function.

---

Nitpick comments:
In `@src/lib/onboard.ts`:
- Around line 9320-9341: This change touches core onboarding preset
reconciliation (applyPresets, applyPreset, waitForPolicyMutation, newlySelected,
accessByName in src/lib/onboard.ts); run the full onboarding E2E matrix (all
sandbox lifecycle/provider/channel/onboarding permutations) before merging to
catch regressions—execute the project's recommended onboarding E2E jobs
(including sandbox creation, preset application, provider/channel flows, and
cleanup) against this branch and report any failures, re-run locally if flaky,
and only merge after green runs and any failures are fixed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 447da395-9749-491a-84e7-270621c7f897

📥 Commits

Reviewing files that changed from the base of the PR and between 9bf3f8d and b701032.

📒 Files selected for processing (6)
  • src/lib/onboard.ts
  • src/lib/policy/index.ts
  • test/install-preflight.test.ts
  • test/onboard-preset-diff.test.ts
  • test/policies.test.ts
  • test/policy-tiers-onboard.test.ts

Comment thread src/lib/onboard.ts Outdated
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 14, 2026

E2E Advisor Recommendation

Required E2E: cloud-onboard-e2e, network-policy-e2e, double-onboard-e2e
Optional E2E: cloud-e2e, onboard-resume-e2e, onboard-repair-e2e, hermes-e2e

Dispatch hint: cloud-onboard-e2e,network-policy-e2e,double-onboard-e2e

Auto-dispatched E2E: cloud-onboard-e2e, network-policy-e2e, double-onboard-e2e via nightly-e2e.yaml at 420f28dc913aa7cd543f3f54b030531eaf6f4bb7

Workflow run

Full advisor summary

Pi Semantic E2E Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • cloud-onboard-e2e: This job onboards non-interactively with NEMOCLAW_POLICY_MODE=custom and NEMOCLAW_POLICY_PRESETS=npm,pypi — exactly the all-built-in additions case that now hits the new batched applyPresets() path in syncPresetSelection. It is the most direct end-to-end exercise of the refactor.
  • network-policy-e2e: Validates deny-by-default, whitelist behavior, live policy-add, and hot-reload after onboarding with restricted tier. The new applyPresets batch path produces the merged YAML now sent to openshell policy set, so a regression in merge/order would surface as broken egress controls.
  • double-onboard-e2e: Re-onboarding is the regression surface for [All Platforms][Regression] presets do not apply correctly #2177 (narrow selection must remove deselected presets) and the additive-suggested behavior preserved in this refactor. The second onboard re-runs syncPresetSelection against a sandbox with already-applied presets.

Optional E2E

  • cloud-e2e: Full cloud onboard + inference smoke. Provides a broad guard that the onboarding step ordering still completes when the policy preset step uses the new batched path.
  • onboard-resume-e2e: Exercises the resumed-onboard path that feeds selectedPresets into setupPoliciesWithSelection — the same function whose preset-application call site moved to the new policy-preset-sync module.
  • onboard-repair-e2e: Repair flow re-enters onboard after partial failure; confirms the extracted policy-preset-sync module loads correctly via require() in the production bundle.
  • hermes-e2e: Hermes onboarding goes through the same setupPoliciesWithSelection / syncPresetSelection path with an agent-specific permissive policy; useful confidence that the extraction did not regress the multi-agent code path.

New E2E recommendations

  • network-policy (medium): The new applyPresets() path submits a single merged YAML for many built-ins at once. Existing E2E coverage applies one preset at a time via nemoclaw policy-add. There is no end-to-end test that asserts onboard with a multi-preset NEMOCLAW_POLICY_PRESETS list (e.g. npm,pypi,huggingface,brew) results in exactly one openshell policy set call AND that egress to all four ecosystems is allowed afterwards.
    • Suggested test: test/e2e/test-onboard-batched-presets.sh: non-interactive onboard with NEMOCLAW_POLICY_MODE=custom and NEMOCLAW_POLICY_PRESETS=npm,pypi,huggingface,brew; assert (a) gateway logs show a single policy version bump, (b) nemoclaw <name> policy show lists all four presets, (c) curl to one host per preset succeeds from inside the sandbox.

Dispatch hint

  • Workflow: nightly-e2e.yaml
  • jobs input: cloud-onboard-e2e,network-policy-e2e,double-onboard-e2e

@github-actions
Copy link
Copy Markdown
Contributor

Selective E2E Results — ✅ All requested jobs passed

Run: 25842822871
Target ref: b701032b0bc6ba0f9e9406761e2992f6a8c483de
Workflow ref: main
Requested jobs: cloud-onboard-e2e,network-policy-e2e,cloud-e2e,onboard-resume-e2e,double-onboard-e2e
Summary: 4 passed, 0 failed, 0 skipped

Job Result
cloud-e2e ✅ success
cloud-onboard-e2e ✅ success
double-onboard-e2e ⚠️ cancelled
network-policy-e2e ✅ success
onboard-resume-e2e ✅ success

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/lib/onboard.ts (1)

69-71: Run the onboarding E2Es for the extracted preset-sync path.

This import reroutes a core onboarding step through the new helper, so it’s worth exercising the recommended onboarding suites before merge, especially cloud-e2e and sandbox-operations-e2e, plus a messaging flow if preset-driven channels are part of the scenario.

As per coding guidelines, src/lib/onboard.ts: "This file contains core onboarding logic. Changes here affect the full sandbox creation and configuration flow."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard.ts` around lines 69 - 71, The new import of
syncPresetSelection from ./onboard/policy-preset-sync changes a core onboarding
step and must be validated by running the onboarding E2E suites; run the
cloud-e2e and sandbox-operations-e2e test suites and exercise any preset-driven
messaging flows to ensure syncPresetSelection behaves correctly during full
sandbox creation and configuration, verify no regressions in src/lib/onboard.ts
onboarding paths, and report/fix any failures in the policy-preset-sync module
(syncPresetSelection) before merging.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/onboard/policy-preset-sync.ts`:
- Around line 54-67: The current logic batches all built-in presets up front
which reorders merges relative to the original sequential application; change it
so batching only happens when every newlySelected preset is built-in, otherwise
preserve original target order by iterating newlySelected and applying each
preset in sequence. Concretely, in the accessByName === false branch use
policies.listPresets() to compute builtInPresetNames as now, then if
builtInNewlySelected.length>0 && remainingNewlySelected.length===0 call
waitForPolicyMutation with policies.applyPresets(sandboxName,
builtInNewlySelected); otherwise loop over newlySelected and for each name call
waitForPolicyMutation(`applyPreset(${name})`, () =>
policies.applyPreset(sandboxName, name)) so built-in and custom presets are
applied in the original order.

---

Nitpick comments:
In `@src/lib/onboard.ts`:
- Around line 69-71: The new import of syncPresetSelection from
./onboard/policy-preset-sync changes a core onboarding step and must be
validated by running the onboarding E2E suites; run the cloud-e2e and
sandbox-operations-e2e test suites and exercise any preset-driven messaging
flows to ensure syncPresetSelection behaves correctly during full sandbox
creation and configuration, verify no regressions in src/lib/onboard.ts
onboarding paths, and report/fix any failures in the policy-preset-sync module
(syncPresetSelection) before merging.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6b28286a-ad4b-4f48-829f-77b7ded1b636

📥 Commits

Reviewing files that changed from the base of the PR and between b701032 and 4c07771.

📒 Files selected for processing (2)
  • src/lib/onboard.ts
  • src/lib/onboard/policy-preset-sync.ts

Comment thread src/lib/onboard/policy-preset-sync.ts
@github-actions
Copy link
Copy Markdown
Contributor

Selective E2E Results — ✅ All requested jobs passed

Run: 25843302449
Target ref: 4c077712f93b1e514977a6fccadcb2dfde74a90e
Workflow ref: main
Requested jobs: cloud-onboard-e2e,network-policy-e2e,double-onboard-e2e
Summary: 3 passed, 0 failed, 0 skipped

Job Result
cloud-onboard-e2e ✅ success
double-onboard-e2e ✅ success
network-policy-e2e ✅ success

@github-actions
Copy link
Copy Markdown
Contributor

Selective E2E Results — ✅ All requested jobs passed

Run: 25844327975
Target ref: e9a9ef008044e8be2ffd8013dd55f4cc57ad4230
Workflow ref: main
Requested jobs: cloud-onboard-e2e,network-policy-e2e,double-onboard-e2e,onboard-repair-e2e
Summary: 4 passed, 0 failed, 0 skipped

Job Result
cloud-onboard-e2e ✅ success
double-onboard-e2e ✅ success
network-policy-e2e ✅ success
onboard-repair-e2e ✅ success

@github-actions
Copy link
Copy Markdown
Contributor

Selective E2E Results — ✅ All requested jobs passed

Run: 25846282461
Target ref: 420f28dc913aa7cd543f3f54b030531eaf6f4bb7
Workflow ref: main
Requested jobs: cloud-onboard-e2e,network-policy-e2e,double-onboard-e2e
Summary: 3 passed, 0 failed, 0 skipped

Job Result
cloud-onboard-e2e ✅ success
double-onboard-e2e ✅ success
network-policy-e2e ✅ success

@cv cv added the v0.0.42 Release target label May 14, 2026
@cv cv merged commit 364f73c into main May 14, 2026
28 checks passed
@miyoungc miyoungc mentioned this pull request May 14, 2026
12 tasks
miyoungc added a commit that referenced this pull request May 14, 2026
## Summary
Refreshes the NemoClaw documentation for the local `main` changes
included in the 0.0.42 release. The update adds release notes, updates
the affected user-facing setup and troubleshooting pages, bumps docs
metadata to 0.0.42, and regenerates the matching user skills.

## Changes
- #3537 -> `docs/reference/commands.md`,
`docs/reference/troubleshooting.md`: Documented host-level status
fields, cloudflared state-specific recovery hints, and Local Ollama auth
proxy status diagnostics.
- #3454 -> `docs/get-started/prerequisites.md`,
`docs/get-started/quickstart.md`: Documented macOS Docker-driver
onboarding and removed the expectation that standard macOS setup needs a
VM driver helper.
- #3514 -> `docs/inference/use-local-inference.md`: Documented
compatible-endpoint retry behavior for reasoning-only smoke responses.
- #3448 -> `docs/reference/commands.md`,
`docs/manage-sandboxes/messaging-channels.md`: Documented canonical
channel names and policy preset hints after `channels add`.
- #3520 -> `docs/about/release-notes.md`: Captured clearer GPU recovery
and uninstall wording in the 0.0.42 release notes.
- #3313 -> `docs/get-started/quickstart.md`,
`docs/reference/troubleshooting.md`: Documented stronger dashboard port
detection and rollback when a forward cannot start.
- #3502 -> `docs/about/release-notes.md`: Captured batched onboarding
policy preset application in the 0.0.42 release notes.
- #3505 -> `docs/reference/troubleshooting.md`: Documented the top-level
Colima socket path.
- #3421 -> `docs/about/release-notes.md`: Captured idempotent installer
shim logging in the 0.0.42 release notes.
- Updated `docs/project.json`, `docs/versions1.json`, and regenerated
`.agents/skills/nemoclaw-user-*` outputs.

## Type of Change
- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Verification
- [ ] `npx prek run --all-files` passes
- [ ] `npm test` passes
- [ ] Tests added or updated for new or changed behavior
- [x] No secrets, API keys, or credentials committed
- [x] Docs updated for user-facing behavior changes
- [x] `make docs` builds without warnings (doc changes only)
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Miyoung Choi <miyoungc@nvidia.com>

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes - v0.0.42

* **Documentation**
  * Enhanced macOS onboarding guidance for Docker gateway setup
  * Improved dashboard port conflict handling with automatic rollback
* Better local Ollama inference diagnostics and authentication proxy
checks
  * Clarified status command output and recovery procedures
  * Refined messaging channel setup documentation

* **Chores**
  * Version bump to 0.0.42

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/NVIDIA/NemoClaw/pull/3540)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants