Skip to content

fix: re-assert the HTTP boot device during NIC-mode boot-order setup#2950

Merged
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-2947
Jun 28, 2026
Merged

fix: re-assert the HTTP boot device during NIC-mode boot-order setup#2950
chet merged 1 commit into
NVIDIA:mainfrom
chet:gh-issue-2947

Conversation

@chet

@chet chet commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

SetBootOrder now re-asserts the HTTP boot device before reordering, so a NIC-mode host's boot setup stays solid even when the BlueField NIC briefly drops off Redfish across a reboot. The BIOS-setup phase already does this on recovery -- this gives SetBootOrder the same small touch, rounding out the zero-DPU boot-interface workflow.

The motivating case: on a reboot the BlueField can de-enumerate and HttpDev1 reverts to the onboard default. set_boot_order_dpu_first only reorders the boot options it finds, so the "HTTP Device 1" option would be gone and is_boot_order_setup could never pass on retry. Re-running machine_setup (by interface id) at the top of the SetBootOrder attempt re-enables it -- the existing RebootHost restart applies it in the same reboot, so no extra reboot, and it's a no-op when HttpDev1 is already set.

Tests added!

This supports #2947 and #2821.

@chet
chet requested a review from a team as a code owner June 28, 2026 03:33
@coderabbitai

coderabbitai Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

The PR re-runs machine_setup before SetBootOrder boot-order reconfiguration, adds Redfish simulator state for HttpDev1 de-enumeration and recovery, extends machine-state tests for zero-DPU hosts, and adds a documented Site Explorer status gauge.

Changes

SetBootOrder HttpDev1 re-assert fix

Layer / File(s) Summary
Redfish simulator state tracking
crates/redfish/src/libredfish/test_support.rs
RedfishSimHostState gains http_dev1_enabled and per-host boot-order readiness; default initialization sets the fields; set_http_dev1_reverted() clears the flag for known hosts; machine_setup restores it; set_boot_order_dpu_first reads it when reporting setup.
Handler reasserts machine_setup before boot order
crates/machine-controller/src/handler.rs
set_host_boot_order now calls machine_setup on the resolved boot interface before set_boot_order_dpu_first_and_handle_no_dpu_error, mapping setup errors and ignoring the returned job id.
Regression tests for recovery and idempotency
crates/api-core/src/tests/machine_states.rs
Adds zero-DPU test helpers, a SetBootOrder-stuck driver, a NIC MAC lookup, an action-order assertion, and two sqlx_test cases covering de-enumerated and healthy hosts.

Site Explorer metric documentation

Layer / File(s) Summary
Metric row update
docs/observability/core_metrics.md
The core metrics table gains a row for carbide_site_explorer_last_run_status.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the core change: re-asserting the HTTP boot device during NIC-mode boot-order setup.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The description accurately matches the implemented SetBootOrder, Redfish simulation, and test changes.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@chet

chet commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai PTAL, thanks!

@coderabbitai

coderabbitai Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

@chet Reviewing this PR now. I’ll focus on correctness, retry behavior around Redfish/NIC-mode recovery, and the added test coverage.

(•ᴗ•)🐇

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/api-core/src/tests/machine_states.rs (1)

3009-3019: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert the action order and selected boot NIC.

These tests only prove that MachineSetup and SetBootOrderDpuFirst happened at some point. They would still pass if machine_setup ran after the reorder, or if it was called without the resolved interface, which is the main contract this PR is adding. Please assert MachineSetup precedes SetBootOrderDpuFirst and that its boot_interface_mac matches the zero-DPU boot NIC.

Also applies to: 3045-3059

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/api-core/src/tests/machine_states.rs` around lines 3009 - 3019, The
current assertions in the machine state tests only check that MachineSetup and
SetBootOrderDpuFirst occurred, but not their order or the selected boot NIC.
Update the relevant assertions in machine_states.rs around the
SetBootOrderDpuFirst flow so they verify MachineSetup happens before
SetBootOrderDpuFirst and that the MachineSetup payload’s boot_interface_mac
matches the resolved zero-DPU boot NIC; use the existing RedfishSimAction
matching in the test helpers to inspect both actions and their fields.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/redfish/src/libredfish/test_support.rs`:
- Around line 69-99: The new http_dev1_enabled flag is incorrectly stored on the
shared RedfishSimState, causing one host’s de-enumeration or machine_setup to
affect all hosts. Move this state into RedfishSimHostState and update the
relevant reads/writes in RedfishSim and any helper methods to access it via
state.hosts[&self._host]. Keep the default behavior the same per host, but
ensure each simulated host maintains its own HttpDev1 enabled state.

---

Nitpick comments:
In `@crates/api-core/src/tests/machine_states.rs`:
- Around line 3009-3019: The current assertions in the machine state tests only
check that MachineSetup and SetBootOrderDpuFirst occurred, but not their order
or the selected boot NIC. Update the relevant assertions in machine_states.rs
around the SetBootOrderDpuFirst flow so they verify MachineSetup happens before
SetBootOrderDpuFirst and that the MachineSetup payload’s boot_interface_mac
matches the resolved zero-DPU boot NIC; use the existing RedfishSimAction
matching in the test helpers to inspect both actions and their fields.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b8520c86-d673-41ab-a313-332e58ecef87

📥 Commits

Reviewing files that changed from the base of the PR and between 87a5337 and 09b9b0c.

📒 Files selected for processing (3)
  • crates/api-core/src/tests/machine_states.rs
  • crates/machine-controller/src/handler.rs
  • crates/redfish/src/libredfish/test_support.rs

Comment thread crates/redfish/src/libredfish/test_support.rs Outdated
@github-actions

Copy link
Copy Markdown

🔍 Container Scan Summary

Service Total Critical High Medium Low Other
boot-artifacts-aarch64 3 0 0 3 0 0
boot-artifacts-x86_64 3 0 0 3 0 0
forge-admin-cli-x86_64 285 6 25 103 7 144
machine-validation-runner 748 32 187 272 36 221
machine_validation 748 32 187 272 36 221
machine_validation-aarch64 748 32 187 272 36 221
nvmetal-carbide 748 32 187 272 36 221
TOTAL 3283 134 773 1197 151 1028

Per-CVE detail lives in the per-service grype-* artifacts (JSON + SARIF). Severity counts only — no CVE IDs published here.

@chet

chet commented Jun 28, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Both addressed in d5d8dac9c:

  • http_dev1_enabled moved onto per-host RedfishSimHostState -- one host's de-enum / machine_setup no longer bleeds to others.
  • The tests now assert machine_setup precedes the reorder and that both target the resolved boot NIC.

Thanks!

@coderabbitai

coderabbitai Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

@chet Thanks for the update. The per-host RedfishSimHostState isolation and the stronger ordering/NIC-target assertions are exactly the right shape for this fix. I’ll re-review the latest changes now.

(•ᴗ•)🐇

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/redfish/src/libredfish/test_support.rs`:
- Around line 1264-1269: Keep boot-order readiness isolated per host:
`set_boot_order_setup` is writing a host-specific `http_dev1_enabled` value into
the shared `state.is_boot_order_setup`, but `is_boot_order_setup()` is still
read globally by all clients. Update the test support state so boot-order
readiness is stored and checked per host (using the existing host state around
`http_dev1_enabled` and `is_boot_order_setup()`), and make sure any caller in
`machine-controller` sees the readiness for the current host only. Add or adjust
tests in `test_support.rs` to cover two hosts toggling `HttpDev1` independently.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 33e333b5-0403-4d36-b171-4bc992d2e4ce

📥 Commits

Reviewing files that changed from the base of the PR and between 09b9b0c and d5d8dac.

📒 Files selected for processing (4)
  • crates/api-core/src/tests/machine_states.rs
  • crates/machine-controller/src/handler.rs
  • crates/redfish/src/libredfish/test_support.rs
  • docs/observability/core_metrics.md
✅ Files skipped from review due to trivial changes (1)
  • docs/observability/core_metrics.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • crates/api-core/src/tests/machine_states.rs
  • crates/machine-controller/src/handler.rs

Comment thread crates/redfish/src/libredfish/test_support.rs Outdated
`SetBootOrder` now re-asserts the HTTP boot device before reordering, so a
NIC-mode host's boot setup stays solid even when the BlueField NIC briefly
drops off Redfish across a reboot. The BIOS-setup phase already does this on
recovery -- this gives `SetBootOrder` the same small touch, rounding out the
zero-DPU boot-interface workflow.

The motivating case: on a reboot the BlueField can de-enumerate and `HttpDev1`
reverts to the onboard default. `set_boot_order_dpu_first` only reorders the
boot options it finds, so the "HTTP Device 1" option would be gone and
`is_boot_order_setup` could never pass on retry. Re-running `machine_setup` (by
interface id) at the top of the SetBootOrder attempt re-enables it -- the
existing `RebootHost` restart applies it in the same reboot, so no extra
reboot, and it's a no-op when HttpDev1 is already set.

While here, regenerates `docs/observability/core_metrics.md` -- the docs check
is enforced now and it had drift from a recent site-explorer metric.

Tests added!

This supports NVIDIA#2947

Signed-off-by: Chet Nichols III <chetn@nvidia.com>
@chet
chet merged commit 28cf6eb into NVIDIA:main Jun 28, 2026
58 checks passed
nv-dmendoza pushed a commit that referenced this pull request Jul 23, 2026
…ingestion (#4012)

> [!IMPORTANT]
> This PR cherry-picks the host boot-interface configuration convergence
arc into `release/v2.0`, in merge order:
> - #3139 (`78aa3766a`) -- skip the SetBootOrder re-apply when the boot
config is already set
> - #3186 (`064ce21bc`) -- apply SetBootOrder boot config once per
reboot
> - #3194 (`428135d4f`) -- resolve the boot NIC through one shared step
> - #3369 (`a2ab63f2c`) -- ensure a correct boot config during machine
validation
> - #3454 (`9d80c7fcd`) -- generalized convergence for host boot
interface configuration

## Related issues

This supports #2821
(nvbug 6430706).

## Type of Change

- [ ] **Add** - New feature or capability
- [ ] **Change** - Changes in existing functionality
- [x] **Fix** - Bug fixes
- [ ] **Remove** - Removed features or deprecated functionality
- [ ] **Internal** - Internal changes (refactoring, tests, docs, etc.)

## Breaking Changes

- [ ] **This PR contains breaking changes**

## Testing

- [x] Unit tests added/updated
- [x] Integration tests added/updated
- [ ] Manual testing performed
- [ ] No testing required (docs, internal refactor, etc.)

## Additional Notes

Backports the boot-interface convergence follow-ups that a v2.0 NIC-mode
host needs to clear `PollingBiosSetup`/`BiosSetupFailed` (nvbug 6430706
/ #2821). The base fix -- #2950, re-assert the HTTP boot device during
NIC-mode -- is already in `release/v2.0`; this arc is the refinement set
that lands the boot config by interface ID when the Dell NDF MAC goes
empty, and converges it across host init, assigned setup, DPU
reprovision, and validation.

- **One conflict, in #3454** (`crates/machine-controller/Cargo.toml`
`[dev-dependencies]`): resolved to `carbide-redfish` +
`carbide-test-harness` + `carbide-test-support`, dropping
`carbide-instrument` -- that crate does not exist on `release/v2.0` and
nothing in the picked code references it. All other commits applied
cleanly.
- **libredfish:** no bump needed. The interface-ID boot resolution
requires libredfish >= v0.44.16; `release/v2.0` already pins
**v0.44.21** (`main` is at v0.44.22).
- **#2896 intentionally excluded:** it was closed unmerged (a libredfish
pin bump + test), superseded by #2950, which is already in v2.0.

---------

Signed-off-by: Chet Nichols III <chetn@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants