Skip to content

feat: Managed switch decommissioning - #4679

Open
ericpretzel wants to merge 4 commits into
NVIDIA:mainfrom
ericpretzel:managed-switch-decommissioning
Open

feat: Managed switch decommissioning#4679
ericpretzel wants to merge 4 commits into
NVIDIA:mainfrom
ericpretzel:managed-switch-decommissioning

Conversation

@ericpretzel

Copy link
Copy Markdown
Contributor

Summary

Adds managed-switch decommissioning, including Core API support, state-controller orchestration, Flow integration, and admin CLI commands.

Decommissioning workflow

Preparing
  → FactoryResetNvos
  → VerifyNvosDhcpRelease
      → ForceRestarting
      → WaitingForDhcpAcknowledgement
  → FactoryResetBmc
  → VerifyDhcpRelease
  → Decommissioned
  • Requires the RMS component-manager backend.
  • Resets NVOS to factory defaults through Component Manager.
  • After reset completion, requests an NVOS force restart and immediately suppresses NVOS DHCP.
  • Waits for NVOS DHCP suppression acknowledgement.
  • Resets the switch BMC through Redfish.
  • Suppresses BMC DHCP only after the Redfish reset succeeds.
  • Waits for BMC DHCP suppression acknowledgement before marking the switch decommissioned.

API and CLI

  • Adds the managed-switch decommission request path.
  • Adds the DeleteDecommissionedSwitch RPC and authorization rules.
  • Allows permanent deletion only after the switch reaches Decommissioned.
  • Adds admin CLI commands:
    • nico-admin-cli managed-switch decommission
    • nico-admin-cli managed-switch delete-decommissioned
  • Adds generated CLI reference pages for both commands.
  • Adds the database flag used to request decommissioning.

Flow integration

  • Adds NVSwitch decommission control and status support to the NICo component-manager implementation.
  • Converts Core’s serialized switch-controller state into Flow’s generic decommission status vocabulary.
  • Preserves polling behavior for in-progress states and reports Decommissioned on completion.

Validation

I have no managed switch in my dev environment so this code path is untested. Will be able to test in other sites that do have racks once this is merged. It has very similar logic to that of managed host decommissioning which does successfully work in my dev environment, so hopefully it will need minimal changes to become ready.

Related issues

Closes #3824

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

@ericpretzel
ericpretzel requested review from a team and polarweasel as code owners August 6, 2026 17:48
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d3942766-be53-4329-bf54-9d2832efad1f

📥 Commits

Reviewing files that changed from the base of the PR and between 4643573 and b98027c.

📒 Files selected for processing (5)
  • crates/api-db/migrations/20260806171754_switch_decommission_requested.sql
  • crates/secrets/src/forge_vault.rs
  • rest-api/flow/internal/nicoapi/grpc.go
  • rest-api/flow/internal/task/componentmanager/nvswitch/nico/nico.go
  • rest-api/flow/internal/task/componentmanager/nvswitch/nico/nico_test.go
🚧 Files skipped from review as they are similar to previous changes (3)
  • rest-api/flow/internal/nicoapi/grpc.go
  • rest-api/flow/internal/task/componentmanager/nvswitch/nico/nico_test.go
  • rest-api/flow/internal/task/componentmanager/nvswitch/nico/nico.go

Summary by CodeRabbit

  • New Features
    • Added managed-switch decommissioning with progress through hardware resets and network cleanup.
    • Added permanent deletion for switches that have completed decommissioning.
    • Added corresponding administrative CLI commands and service APIs.
    • Added status reporting for decommissioning progress.
  • Bug Fixes
    • Credential cleanup now handles already-removed Vault entries successfully.
  • Documentation
    • Added usage guides and examples for both managed-switch lifecycle commands.

Walkthrough

Changes

Managed switch decommissioning

Layer / File(s) Summary
Decommissioning contracts and persistence
crates/rpc/proto/forge.proto, rest-api/proto/core/src/v1/nico_nico.proto, crates/api-model/src/switch/mod.rs, crates/api-db/..., crates/rpc/src/model/switch.rs
Adds decommissioning RPCs, controller states, serialization coverage, and the persisted decommission_requested flag.
Controller decommissioning workflow
crates/switch-controller/src/...
Adds state dispatch for NVOS reset, DHCP suppression checks, BMC reset, and the terminal Decommissioned state.
API and CLI operations
crates/api-core/src/..., crates/admin-cli/src/managed_switch/..., crates/secrets/src/forge_vault.rs, docs/manuals/nico-admin-cli/commands/managed-switch/...
Adds authorization, API validation and cleanup, CLI commands, credential deletion handling, command tests, and documentation.
Flow RPC and status integration
rest-api/flow/internal/...
Calls the Core decommission RPC and converts Core controller-state JSON into Flow decommissioning status values.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AdminCLI
  participant CoreAPI
  participant SwitchController
  participant NVOS
  participant BMC
  participant DHCPRecords
  AdminCLI->>CoreAPI: request switch decommission
  CoreAPI->>SwitchController: persist decommission request
  SwitchController->>NVOS: reset and verify NVOS
  NVOS-->>SwitchController: reset and DHCP results
  SwitchController->>BMC: reset BMC
  BMC-->>SwitchController: reset result
  SwitchController->>DHCPRecords: verify DHCP suppression
  SwitchController-->>CoreAPI: mark switch Decommissioned
Loading
sequenceDiagram
  participant AdminCLI
  participant CoreAPI
  participant CredentialStore
  participant Database
  AdminCLI->>CoreAPI: delete decommissioned switch
  CoreAPI->>CredentialStore: delete BMC and NVOS credentials
  CredentialStore-->>CoreAPI: deletion results
  CoreAPI->>Database: remove interfaces, suppression records, and switch
  Database-->>CoreAPI: deletion result
  CoreAPI-->>AdminCLI: return success or error
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 47.92% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the primary change: managed-switch decommissioning.
Description check ✅ Passed The description directly explains the decommissioning workflow, APIs, controller changes, Flow integration, CLI, and database support.
Linked Issues check ✅ Passed The changes implement managed-switch decommissioning states and supporting APIs, orchestration, and deletion flow required by [#3824].
Out of Scope Changes check ✅ Passed The changes remain within the decommissioning feature scope, including credential cleanup behavior required for switch deletion.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🧹 Nitpick comments (5)
crates/api-core/src/tests/switch_state_controller/mod.rs (1)

159-170: 🗄️ Data Integrity & Integration | 🔵 Trivial | 🏗️ Heavy lift

Test the persistent cleanup contract.

The test verifies only removal of the switches row. Add associated machine interfaces, retained boot interfaces, and DHCP suppression records to the fixture. Assert that the deletion RPC removes each record. A partial cleanup currently passes this test.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/api-core/src/tests/switch_state_controller/mod.rs` around lines 159 -
170, Expand the deletion test around delete_decommissioned_switch to seed
associated machine interfaces, retained boot interfaces, and DHCP suppression
records for the fixture switch, then query each repository after the RPC and
assert every record is removed. Keep the existing switches-row assertion and use
the corresponding fixture identifiers to verify complete persistent cleanup.
crates/api-model/src/switch/mod.rs (2)

603-641: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a round-trip case for FactoryResetNvos { job_id: None }.

The table covers the accepted-job form (job_id: Some(...)) but not the pre-submission form. job_id: None is the state the controller persists first, and handle_preparing transitions directly into it, so it is the variant that must survive a restart. Add both directions to lock the wire form for the null case.

💚 Suggested additional scenarios

Serialization table:

+            "decommissioning: NVOS reset not yet submitted" {
+                SwitchControllerState::Decommissioning {
+                    decommissioning_state: SwitchDecommissioningState::FactoryResetNvos {
+                        job_id: None,
+                    },
+                } => Yields(
+                    r#"{"state":"decommissioning","decommissioning_state":{"state":"factoryresetnvos","job_id":null}}"#
+                        .to_string(),
+                ),
+            }
+

Deserialization table:

+            "decommissioning NVOS reset with absent job_id" {
+                r#"{"state":"decommissioning","decommissioning_state":{"state":"factoryresetnvos"}}"# => Yields(
+                    SwitchControllerState::Decommissioning {
+                        decommissioning_state: SwitchDecommissioningState::FactoryResetNvos {
+                            job_id: None,
+                        },
+                    },
+                ),
+            }
+

As per coding guidelines: "Prefer table-driven tests for functions mapping inputs to outputs or errors. Use scenarios! with Outcome for fallible operations".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/api-model/src/switch/mod.rs` around lines 603 - 641, Add table-driven
round-trip scenarios for SwitchDecommissioningState::FactoryResetNvos with
job_id: None in both serialization and deserialization test tables. Assert the
wire representation uses "job_id":null, alongside the existing Some("reset-1")
case, so the pre-submission state is preserved across restarts.

Source: Coding guidelines


495-495: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Consider an SLA for the active decommissioning substates.

Decommissioning is not a steady state. It submits an RMS factory-reset job, force-restarts NVOS, resets the BMC, and then waits for two DHCP suppression acknowledgements. handle_factory_reset_nvos and the two verification handlers return StateHandlerOutcome::wait(...), so a switch can remain in a wait loop indefinitely if the RMS job never settles or the DHCP acknowledgement never arrives. With no_sla(), that stall never surfaces through the SLA signal.

Decommissioned is genuinely terminal, so a blanket SLA would fire falsely. Match on the substate instead and apply an SLA only to the in-flight phases.

♻️ Suggested per-substate SLA
-        SwitchControllerState::Decommissioning { .. } => StateSla::no_sla(),
+        SwitchControllerState::Decommissioning {
+            decommissioning_state,
+        } => match decommissioning_state {
+            SwitchDecommissioningState::Decommissioned => StateSla::no_sla(),
+            _ => StateSla::with_sla(
+                std::time::Duration::from_secs(slas::DECOMMISSIONING),
+                time_in_state,
+            ),
+        },

This requires a new slas::DECOMMISSIONING constant sized for the NVOS reset plus both DHCP acknowledgement waits.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/api-model/src/switch/mod.rs` at line 495, Update the Decommissioning
SLA handling in the switch state mapping to match its substates rather than
returning StateSla::no_sla() for every Decommissioning variant. Add
slas::DECOMMISSIONING for the in-flight phases, including the NVOS reset and
DHCP acknowledgement waits, while keeping the terminal Decommissioned substate
on no_sla().
crates/switch-controller/src/ready.rs (1)

50-59: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Log the entry into decommissioning.

The precedence and the atomicity are correct here: decommissioning outranks maintenance and reprovisioning, and the flag is cleared in the same transaction that carries the transition, so the request can neither be lost nor consumed twice.

The branch emits no log, unlike the maintenance branch at lines 62-66 and the reprovisioning branches at lines 74-77 and 91-95. Entering decommissioning starts an irreversible sequence of factory resets. Add a structured log line so operators can correlate the transition with the destructive calls that follow.

♻️ Suggested log line
     if state.decommission_requested {
+        tracing::info!(
+            switch_id = %switch_id,
+            "Switch decommissioning requested; transitioning to Decommissioning"
+        );
         let mut txn = ctx.services.db_pool.begin().await?;
         db_switch::clear_decommission_requested(&mut txn, *switch_id).await?;

As per coding guidelines: "Emit logfmt-compatible structured logs. Tracing messages must be stable human-readable string literals; record dynamic values as structured fields instead of interpolating them."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switch-controller/src/ready.rs` around lines 50 - 59, In the
decommission_requested branch of the ready-state handler, add a structured log
entry immediately before starting the transaction and transitioning to
Decommissioning. Use a stable human-readable message literal and record the
switch identifier as a structured field, matching the logging pattern used by
the maintenance and reprovisioning branches.

Source: Coding guidelines

crates/switch-controller/src/decommissioning.rs (1)

133-189: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add table-driven coverage for the factory-reset job-status mapping.

This module has no test module. The integration test in crates/api-core/src/tests/switch_state_controller/mod.rs covers only Ready -> Preparing -> FactoryResetNvos { job_id: None } and the post-Decommissioned deletion. The SwitchFactoryResetState match at lines 150-169 is the decision point that determines whether a decommission progresses, waits, or demands manual intervention, and it is untested. The pull request notes the managed-switch path has not been exercised in a development environment, which raises the value of this coverage.

The mapping from SwitchFactoryResetState plus an optional error string to a StateHandlerOutcome or StateHandlerError is a pure decision. Extract it into a small function and cover it with scenarios!, so the three branches and the error-suffix formatting are locked without a database or an RMS backend.

The MissingData paths in handle_factory_reset_bmc and handle_verify_dhcp_release are similarly cheap to cover once the switch fixture is available.

As per path instructions: "Prefer findings about behavior, concurrency, resource lifetimes, and missing tests over style-only comments." As per coding guidelines: "Prefer table-driven tests for functions mapping inputs to outputs or errors. Use scenarios! with Outcome for fallible operations."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switch-controller/src/decommissioning.rs` around lines 133 - 189,
Extract the SwitchFactoryResetState match from handle_factory_reset_nvos into a
pure decision function that maps state and optional error text to the existing
outcome or error types, preserving pending, completed, failed, and suffix
formatting behavior. Add table-driven scenarios! coverage using Outcome for all
branches, including failed jobs with and without error text, without requiring
database or RMS services. Also cover the MissingData paths in
handle_factory_reset_bmc and handle_verify_dhcp_release if the existing switch
fixture supports them.

Sources: Coding guidelines, Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/api-core/src/auth/internal_rbac_rules.rs`:
- Around line 302-303: Update the DeleteDecommissionedSwitch permission entry in
the RBAC rule definitions to grant access only to ForgeAdminCLI. Remove Flow
from its caller list while leaving the DecommissionSwitch permission unchanged.

In `@crates/api-core/src/handlers/switch.rs`:
- Around line 385-396: Make the credential cleanup in the switch deletion flow
idempotent by treating a not-found response from Vault metadata deletion as
success, while still propagating other errors; update
delete_bmc_root_credentials_by_mac and the NVOS credential deletion path as
needed. Add a retry test covering partial cleanup, verifying the switch row and
remaining NVOS credential are deleted successfully when the BMC secret is
already absent.

In `@crates/switch-controller/src/decommissioning.rs`:
- Around line 179-188: The factory-reset submission flow around
batch_reset_switch_factory_default must prevent duplicate RMS resets when state
persistence fails after acceptance. Add switch-keyed idempotency or job
reconciliation that records and recovers the submission intent, preserving
OperationOutcomeUnknown semantics so retries do not resubmit an uncertain
operation. Add a failure-path test verifying that only one RMS reset is
submitted.
- Around line 112-131: Centralize the RMS backend validation currently performed
in handle_preparing into a shared helper, preserving the existing
missing-backend and non-RMS InvalidState errors. Invoke that helper from
handle_preparing, handle_factory_reset_nvos, handle_verify_nvos_dhcp_release,
and decommission_switch so persisted states and direct decommission operations
are validated before proceeding; if needed, add a typed NvSwitchManager
discriminator instead of duplicating raw name checks.
- Around line 191-220: The decommissioning flow must remain usable after
factory-reset credentials change. In handle_verify_nvos_dhcp_release
(crates/switch-controller/src/decommissioning.rs:191-220), defer full endpoint
resolution until ForceRestarting and use a MAC-only lookup or persisted MAC
while awaiting DHCP; ensure power_control receives post-reset credentials or
force-restarts before reset. In the retry path at
crates/switch-controller/src/decommissioning.rs:277-286, make retries after
bmc_reset_to_defaults idempotent and reset-aware so DHCP suppression or
transition-commit failures do not recreate the client with stale stored BMC
credentials.

In
`@docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-decommission.md`:
- Around line 15-17: Update the managed-switch decommission documentation at
docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-decommission.md:15-17
to describe Ready-state and RMS-backend prerequisites, request persistence,
asynchronous progression, terminal outcome, and failure behavior; update
docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-delete-decommissioned.md:16-18
to document the terminal Decommissioned prerequisite, irreversible deletion,
removed managed records, retry behavior, and unsupported paths, then regenerate
the CLI references from the canonical source.

In `@rest-api/flow/internal/nicoapi/grpc.go`:
- Around line 570-573: Update the error message returned by the
DecommissionSwitch call to begin with lower-case text, while preserving the
switchID context and wrapped err value.

In `@rest-api/flow/internal/task/componentmanager/nvswitch/nico/nico.go`:
- Around line 622-628: Update the state-formatting logic around the JSON
unmarshal to return raw when state.DecommissioningState.State is empty, before
constructing the "Decommissioning/" result. Preserve the existing
"Decommissioned" and non-empty decommissioning-state behavior, and add coverage
for {"state":"decommissioning"}.

---

Nitpick comments:
In `@crates/api-core/src/tests/switch_state_controller/mod.rs`:
- Around line 159-170: Expand the deletion test around
delete_decommissioned_switch to seed associated machine interfaces, retained
boot interfaces, and DHCP suppression records for the fixture switch, then query
each repository after the RPC and assert every record is removed. Keep the
existing switches-row assertion and use the corresponding fixture identifiers to
verify complete persistent cleanup.

In `@crates/api-model/src/switch/mod.rs`:
- Around line 603-641: Add table-driven round-trip scenarios for
SwitchDecommissioningState::FactoryResetNvos with job_id: None in both
serialization and deserialization test tables. Assert the wire representation
uses "job_id":null, alongside the existing Some("reset-1") case, so the
pre-submission state is preserved across restarts.
- Line 495: Update the Decommissioning SLA handling in the switch state mapping
to match its substates rather than returning StateSla::no_sla() for every
Decommissioning variant. Add slas::DECOMMISSIONING for the in-flight phases,
including the NVOS reset and DHCP acknowledgement waits, while keeping the
terminal Decommissioned substate on no_sla().

In `@crates/switch-controller/src/decommissioning.rs`:
- Around line 133-189: Extract the SwitchFactoryResetState match from
handle_factory_reset_nvos into a pure decision function that maps state and
optional error text to the existing outcome or error types, preserving pending,
completed, failed, and suffix formatting behavior. Add table-driven scenarios!
coverage using Outcome for all branches, including failed jobs with and without
error text, without requiring database or RMS services. Also cover the
MissingData paths in handle_factory_reset_bmc and handle_verify_dhcp_release if
the existing switch fixture supports them.

In `@crates/switch-controller/src/ready.rs`:
- Around line 50-59: In the decommission_requested branch of the ready-state
handler, add a structured log entry immediately before starting the transaction
and transitioning to Decommissioning. Use a stable human-readable message
literal and record the switch identifier as a structured field, matching the
logging pattern used by the maintenance and reprovisioning branches.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: d06300a0-bf27-4d30-9710-48ad41d1c800

📥 Commits

Reviewing files that changed from the base of the PR and between b5eadb0 and 4643573.

⛔ Files ignored due to path filters (2)
  • rest-api/proto/core/gen/v1/nico_nico.pb.go is excluded by !**/*.pb.go, !**/gen/**, !rest-api/**/*.pb.go
  • rest-api/proto/core/gen/v1/nico_nico_grpc.pb.go is excluded by !**/*.pb.go, !**/gen/**, !rest-api/**/*.pb.go, !rest-api/**/*_grpc.pb.go
📒 Files selected for processing (29)
  • crates/admin-cli/src/managed_switch/decommission/args.rs
  • crates/admin-cli/src/managed_switch/decommission/cmd.rs
  • crates/admin-cli/src/managed_switch/decommission/mod.rs
  • crates/admin-cli/src/managed_switch/delete_decommissioned/args.rs
  • crates/admin-cli/src/managed_switch/delete_decommissioned/cmd.rs
  • crates/admin-cli/src/managed_switch/delete_decommissioned/mod.rs
  • crates/admin-cli/src/managed_switch/mod.rs
  • crates/admin-cli/src/managed_switch/tests.rs
  • crates/api-core/src/api.rs
  • crates/api-core/src/auth/internal_rbac_rules.rs
  • crates/api-core/src/handlers/switch.rs
  • crates/api-core/src/tests/switch_state_controller/mod.rs
  • crates/api-db/migrations/20260806120000_switch_decommission_requested.sql
  • crates/api-db/src/switch.rs
  • crates/api-model/src/switch/mod.rs
  • crates/rpc/proto/forge.proto
  • crates/rpc/src/model/switch.rs
  • crates/switch-controller/src/decommissioning.rs
  • crates/switch-controller/src/handler.rs
  • crates/switch-controller/src/io.rs
  • crates/switch-controller/src/lib.rs
  • crates/switch-controller/src/ready.rs
  • docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-decommission.md
  • docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-delete-decommissioned.md
  • docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch.md
  • rest-api/flow/internal/nicoapi/grpc.go
  • rest-api/flow/internal/task/componentmanager/nvswitch/nico/nico.go
  • rest-api/flow/internal/task/componentmanager/nvswitch/nico/nico_test.go
  • rest-api/proto/core/src/v1/nico_nico.proto

Comment on lines +302 to +303
x.perm("DecommissionSwitch", vec![ForgeAdminCLI, Flow]);
x.perm("DeleteDecommissionedSwitch", vec![ForgeAdminCLI, Flow]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

rg -n -C 4 '\.DeleteDecommissionedSwitch\s*\(' rest-api/flow
rg -n -C 4 'DeleteDecommissionedSwitch' crates rest-api/flow

Repository: NVIDIA/infra-controller

Length of output: 161


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- permission definitions ---'
sed -n '270,320p' crates/api-core/src/auth/internal_rbac_rules.rs

printf '%s\n' '--- exact symbol references ---'
rg -n -C 5 'DeleteDecommissionedSwitch|DecommissionSwitch' crates rest-api/flow || true

printf '%s\n' '--- Flow authorization and client method references ---'
rg -n -C 4 'Flow|flow' crates/api-core/src/auth/internal_rbac_rules.rs rest-api/flow || true

Repository: NVIDIA/infra-controller

Length of output: 50380


🏁 Script executed:

#!/usr/bin/env bash
set -euo pipefail

printf '%s\n' '--- exact deletion symbol matches ---'
rg -n -F 'DeleteDecommissionedSwitch' crates rest-api/flow || true

printf '%s\n' '--- exact decommission symbol matches ---'
rg -n -F 'DecommissionSwitch' crates rest-api/flow || true

printf '%s\n' '--- permission block ---'
sed -n '294,308p' crates/api-core/src/auth/internal_rbac_rules.rs

printf '%s\n' '--- generated/API definitions that may expose the RPC ---'
rg -n -F 'DeleteDecommissionedSwitch' . --glob '!target/**' --glob '!node_modules/**' --glob '!*.lock' || true

Repository: NVIDIA/infra-controller

Length of output: 9727


Restrict permanent switch deletion to ForgeAdminCLI.

Flow calls DecommissionSwitch but has no DeleteDecommissionedSwitch caller. Remove Flow from this permission.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/api-core/src/auth/internal_rbac_rules.rs` around lines 302 - 303,
Update the DeleteDecommissionedSwitch permission entry in the RBAC rule
definitions to grant access only to ForgeAdminCLI. Remove Flow from its caller
list while leaving the DecommissionSwitch permission unchanged.

Source: Path instructions

Comment thread crates/api-core/src/handlers/switch.rs
Comment on lines +112 to +131
fn handle_preparing(
switch_id: &SwitchId,
ctx: &StateHandlerContext<'_, SwitchStateHandlerContextObjects>,
) -> Result<StateHandlerOutcome<SwitchControllerState>, StateHandlerError> {
let component_manager = ctx.services.component_manager.as_ref().ok_or_else(|| {
StateHandlerError::InvalidState(format!(
"switch {switch_id} requires the RMS component-manager backend for decommissioning"
))
})?;
if component_manager.nv_switch.name() != "rms" {
return Err(StateHandlerError::InvalidState(format!(
"switch {switch_id} requires the RMS component-manager backend for decommissioning; configured backend is {}",
component_manager.nv_switch.name()
)));
}

Ok(StateHandlerOutcome::transition(decommissioning(
SwitchDecommissioningState::FactoryResetNvos { job_id: None },
)))
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Locate the nv_switch backend trait and any typed backend discriminant.
set -euo pipefail

fd -t f -e rs . crates/component-manager --exec ast-grep outline {} --items all \; 2>/dev/null | head -100

# Find the `name()` definition on the NvSwitch manager trait and its implementors.
rg -nP --type=rust -C6 'fn\s+name\s*\(\s*&self\s*\)\s*->' crates/component-manager

# Find every site that compares the backend name to a literal.
rg -nP --type=rust -C3 'nv_switch\s*\.\s*name\s*\(\s*\)'

Repository: NVIDIA/infra-controller

Length of output: 16784


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- Backend enum and trait ---'
cat -n crates/component-manager/src/nv_switch_manager.rs | sed -n '1,45p;145,170p'
printf '%s\n' '--- RMS backend implementation ---'
cat -n crates/component-manager/src/rms.rs | sed -n '1438,1460p'
printf '%s\n' '--- Decommissioning handlers ---'
cat -n crates/switch-controller/src/decommissioning.rs | sed -n '90,225p'
printf '%s\n' '--- API handler ---'
cat -n crates/api-core/src/handlers/switch.rs | sed -n '245,280p'
printf '%s\n' '--- ComponentManager ownership and construction ---'
rg -n -C4 --type rust 'component_manager|ComponentManagerConfig|NvSwitchBackend' crates/switch-controller crates/api-core crates/component-manager | head -240

Repository: NVIDIA/infra-controller

Length of output: 30893


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- Component manager type and construction ---'
ast-grep outline crates/component-manager/src/component_manager.rs --items all
rg -n -C5 --type rust 'pub struct ComponentManager|struct ComponentManager|nv_switch:' crates/component-manager/src/component_manager.rs crates/api-core crates/switch-controller

printf '%s\n' '--- All NV-switch backend-name comparisons and constants ---'
rg -n -C3 --type rust 'name\(\)\s*[!=]=|BACKEND_NAME|Backend::Rms|NvSwitchBackend' crates/switch-controller crates/api-core crates/component-manager/src

printf '%s\n' '--- Decommission state persistence and handler context setup ---'
rg -n -C5 --type rust 'FactoryResetNvos|SwitchDecommissioningState|SwitchStateHandlerContextObjects|component_manager:' crates/switch-controller crates/api-core crates/state-controller crates/model 2>/dev/null | head -300

Repository: NVIDIA/infra-controller

Length of output: 50379


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- Context ownership ---'
cat -n crates/switch-controller/src/context.rs | sed -n '1,140p'

printf '%s\n' '--- State serialization and persisted controller state ---'
rg -n -C4 --type rust 'controller_state|serde_json|SwitchControllerState|decommissioning_state' crates/model crates/db crates/switch-controller | head -220

printf '%s\n' '--- Decommissioning tests and backend setup ---'
rg -n -C6 --type rust 'handle_preparing|handle_factory_reset_nvos|decommission|MockNvSwitchManager|name\(\)' crates/switch-controller/src crates/api-core/src/tests | head -260

Repository: NVIDIA/infra-controller

Length of output: 20487


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- Factory-reset implementations ---'
rg -n -C8 --type rust 'fn batch_reset_switch_factory_default|batch_reset_switch_factory_default' crates/component-manager/src/nsm.rs crates/component-manager/src/mock.rs crates/component-manager/src/rms.rs crates/component-manager/src/nv_switch_manager.rs

printf '%s\n' '--- State model location and persistence fields ---'
fd -t f -e rs . | rg '(^|/)(switch|model|db)(/|_)' | head -80
rg -n -C4 --type rust 'pub enum SwitchControllerState|SwitchDecommissioningState|controller_state.*json|controller_state.*value' . | head -220

Repository: NVIDIA/infra-controller

Length of output: 43465


Centralize RMS backend validation before each decommissioning operation.

handle_preparing checks name() == "rms", but handle_factory_reset_nvos and handle_verify_nvos_dhcp_release check only for presence. A persisted FactoryResetNvos state can therefore resume with a non-RMS backend and reach an operation that the backend does not support. Reuse one shared backend check in these handlers and decommission_switch. The existing Backend enum is not exposed by NvSwitchManager; add a typed discriminator if the raw "rms" comparison is removed.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switch-controller/src/decommissioning.rs` around lines 112 - 131,
Centralize the RMS backend validation currently performed in handle_preparing
into a shared helper, preserving the existing missing-backend and non-RMS
InvalidState errors. Invoke that helper from handle_preparing,
handle_factory_reset_nvos, handle_verify_nvos_dhcp_release, and
decommission_switch so persisted states and direct decommission operations are
validated before proceeding; if needed, add a typed NvSwitchManager
discriminator instead of duplicating raw name checks.

Source: Coding guidelines

Comment on lines +179 to +188
let job_id = component_manager
.nv_switch
.batch_reset_switch_factory_default(&[endpoint], tls_server_domain.as_deref())
.await
.map_err(|error| external_error("failed to submit NVOS factory reset", error))?;
Ok(StateHandlerOutcome::transition(decommissioning(
SwitchDecommissioningState::FactoryResetNvos {
job_id: Some(job_id),
},
)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Inspect the RMS factory-reset submission contract and any job-lookup-by-target API.
set -euo pipefail

rg -nP --type=rust -C10 'fn\s+batch_reset_switch_factory_default' crates

# Look for an API that lists or finds reset jobs for a given switch, which would enable recovery.
rg -nP --type=rust -C5 'fn\s+\w*(list|find|get)\w*(factory_reset|reset_job)\w*\s*\(' crates

Repository: NVIDIA/infra-controller

Length of output: 11284


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- RMS submission implementation ---'
sed -n '1928,1992p' crates/component-manager/src/rms.rs

printf '%s\n' '--- component-manager contract ---'
sed -n '458,520p' crates/component-manager/src/component_manager.rs
sed -n '210,270p' crates/component-manager/src/nv_switch_manager.rs

printf '%s\n' '--- decommissioning handler and state model references ---'
sed -n '125,205p' crates/switch-controller/src/decommissioning.rs
rg -n -C5 'FactoryResetNvos|job_id.*persist|Once accepted|transition\(' crates/switch-controller crates/api-model crates -g '*.rs' | head -n 240

Repository: NVIDIA/infra-controller

Length of output: 31773


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- reset request and response types ---'
rg -n -C8 'BatchResetSwitchFactoryDefault(Request|Response)|batch_reset_switch_factory_default' crates/component-manager/src crates/api-test-helper/src -g '*.rs'

printf '%s\n' '--- reset-job lookup APIs ---'
rg -n -i -C4 'factory.?reset|reset.?job|operation.?outcome.?unknown|idempot' crates/component-manager/src crates/switch-controller/src -g '*.rs' | head -n 320

Repository: NVIDIA/infra-controller

Length of output: 50379


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- RMS outcome mapping ---'
sed -n '2190,2255p' crates/component-manager/src/rms.rs

printf '%s\n' '--- state outcome persistence path ---'
rg -n -C8 'struct StateHandlerOutcome|enum StateHandlerOutcome|impl.*StateHandlerOutcome|with_txn|outcome.*transition|persist.*state|update.*state' crates -g '*.rs' | head -n 260

printf '%s\n' '--- decommissioning state definition and persistence fields ---'
rg -n -C12 'enum SwitchDecommissioningState|FactoryResetNvos|decommissioning_state' crates -g '*.rs' | head -n 260

Repository: NVIDIA/infra-controller

Length of output: 44749


Prevent automatic resubmission after an uncertain RMS reset submission.

If state persistence fails after RMS accepts the reset, the persisted state still has job_id: None. The next retry submits the destructive operation again. Component Manager explicitly maps this outcome to OperationOutcomeUnknown and forbids automatic resubmission because dispatch may have occurred. Add idempotency or job reconciliation keyed by the switch, with recovery semantics for the submission intent. Add a failure-path test that proves only one RMS reset is submitted.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switch-controller/src/decommissioning.rs` around lines 179 - 188, The
factory-reset submission flow around batch_reset_switch_factory_default must
prevent duplicate RMS resets when state persistence fails after acceptance. Add
switch-keyed idempotency or job reconciliation that records and recovers the
submission intent, preserving OperationOutcomeUnknown semantics so retries do
not resubmit an uncertain operation. Add a failure-path test verifying that only
one RMS reset is submitted.

Comment on lines +191 to +220
async fn handle_verify_nvos_dhcp_release(
switch_id: &SwitchId,
verifying_state: &VerifyNvosDhcpReleaseState,
ctx: &mut StateHandlerContext<'_, SwitchStateHandlerContextObjects>,
) -> Result<StateHandlerOutcome<SwitchControllerState>, StateHandlerError> {
let endpoint = resolve_switch_endpoint(
switch_id,
&ctx.services.db_pool,
&ctx.services.credential_manager,
)
.await?;
match verifying_state {
VerifyNvosDhcpReleaseState::ForceRestarting => {
let component_manager = ctx.services.component_manager.as_ref().ok_or_else(|| {
StateHandlerError::InvalidState(format!(
"switch {switch_id} requires the RMS component-manager backend for decommissioning"
))
})?;
let result = component_manager
.nv_switch
.power_control(std::slice::from_ref(&endpoint), PowerAction::ForceRestart)
.await
.map_err(|error| external_error("failed to force restart NVOS", error))?
.into_iter()
.next()
.ok_or_else(|| {
StateHandlerError::GenericError(eyre::eyre!(
"component manager returned no force-restart result for switch {switch_id}"
))
})?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Determine how each post-reset call authenticates, and whether a cheaper NVOS MAC source exists.
set -euo pipefail

fd -t f 'endpoint.rs' crates/switch-controller --exec cat -n {}
rg -nP --type=rust -C15 'fn\s+resolve_switch_endpoint' crates

rg -nP --type=rust -C10 'fn\s+power_control' crates/component-manager
rg -nP --type=rust -C10 'fn\s+bmc_reset_to_defaults' crates
rg -nP --type=rust -C10 'fn\s+for_bmc_mac' crates

rg -nP --type=rust -C3 '\bnvos_mac\b' crates

Repository: NVIDIA/infra-controller

Length of output: 50379


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- decommissioning relevant ranges ---'
cat -n crates/switch-controller/src/decommissioning.rs | sed -n '1,340p'

echo '--- decommissioning symbols and call sites ---'
rg -n -C5 --type rust \
  'handle_verify_nvos_dhcp_release|VerifyNvosDhcpReleaseState|SwitchFactoryResetState|FactoryResetBmc|create_client|bmc_reset_to_defaults|suppress_dhcp|resolve_switch_endpoint' \
  crates/switch-controller/src/decommissioning.rs

echo '--- RMS NV switch power control ---'
cat -n crates/component-manager/src/rms.rs | sed -n '1440,1585p'

echo '--- NV switch endpoint registration and credential use ---'
cat -n crates/component-manager/src/rms.rs | sed -n '840,935p'
cat -n crates/component-manager/src/nsm.rs | sed -n '55,105p'

echo '--- Redfish client construction and reset-related callers ---'
rg -n -C8 --type rust \
  'fn\s+create_client|create_client\(|RedfishAuth::for_bmc_mac|FactoryResetBmc|bmc_reset_to_defaults' \
  crates/switch-controller crates/redfish crates/component-manager

Repository: NVIDIA/infra-controller

Length of output: 50379


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- NVOS factory reset implementations ---'
rg -n -C12 --type rust \
  'batch_reset_switch_factory_default|get_switch_factory_reset_job_status|factory.?reset|reset_switch_factory' \
  crates/component-manager/src crates/api-core/src

echo '--- RMS request credential serialization and client boundary ---'
rg -n -C10 --type rust \
  'BatchSetPowerStateRequest|batch_set_power_state|credentials_to_rms|build_switch_node_info|RmsApi' \
  crates/component-manager/src/rms.rs crates/component-manager/src

echo '--- Redfish reset implementation and auth flow ---'
rg -n -C15 --type rust \
  'bmc_reset_to_defaults|ResetToDefaults|reset.*defaults|RedfishAuth::Key|CredentialKey|auth_attempts|authenticate' \
  crates/redfish/src crates/switch-controller/src

echo '--- Switch model MAC fields and endpoint-row query helpers ---'
rg -n -C8 --type rust \
  'struct Switch\b|nvos_mac|bmc_info|find_switch_endpoints_by_ids|SwitchEndpointRow' \
  crates/api-model crates/db crates/switch-controller/src

Repository: NVIDIA/infra-controller

Length of output: 50380


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- exact symbol locations ---'
rg -l --type rust 'batch_reset_switch_factory_default|get_switch_factory_reset_job_status' crates/component-manager crates/api-core
rg -n --type rust --max-count 8 'batch_reset_switch_factory_default|get_switch_factory_reset_job_status' crates/component-manager crates/api-core

echo '--- redfish reset symbol locations ---'
rg -l --type rust 'bmc_reset_to_defaults' crates/redfish
rg -n --type rust --max-count 12 'bmc_reset_to_defaults' crates/redfish

echo '--- Redfish auth implementation locations ---'
rg -l --type rust 'impl RedfishClientPool|RedfishAuth::Key|CredentialKey' crates/redfish/src
rg -n --type rust --max-count 20 'RedfishAuth::Key|CredentialKey|create_client' crates/redfish/src/libredfish/implementation.rs crates/redfish/src/libredfish/auth.rs

echo '--- switch model and endpoint query locations ---'
rg -l --type rust 'pub struct Switch\b|struct SwitchEndpointRow|find_switch_endpoints_by_ids' crates/api-model crates/db crates/switch-controller
rg -n --type rust --max-count 12 'pub struct Switch\b|struct SwitchEndpointRow|find_switch_endpoints_by_ids' crates/api-model crates/db crates/switch-controller

echo '--- narrow credential assertions in relevant tests ---'
rg -n --type rust --max-count 20 -C3 \
  'host_endpoint.*credentials|bmc_endpoint.*credentials|nvos_credentials|bmc_credentials|bmc_reset_to_defaults' \
  crates/component-manager/src/rms.rs crates/redfish/src/libredfish/test_support.rs

Repository: NVIDIA/infra-controller

Length of output: 3933


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo '--- RMS reset helpers ---'
cat -n crates/component-manager/src/rms.rs | sed -n '1935,2020p'
cat -n crates/component-manager/src/rms.rs | sed -n '2185,2230p'
cat -n crates/component-manager/src/rms.rs | sed -n '2350,2395p'

echo '--- Redfish client authentication ---'
cat -n crates/redfish/src/libredfish/implementation.rs | sed -n '68,125p'
cat -n crates/redfish/src/libredfish/auth.rs | sed -n '18,38p'

echo '--- switch model fields ---'
cat -n crates/api-model/src/switch/mod.rs | sed -n '1,180p'

echo '--- endpoint row and query definitions ---'
rg -l --type rust 'struct SwitchEndpointRow|find_switch_endpoints_by_ids' .
rg -n --type rust --max-count 20 'struct SwitchEndpointRow|find_switch_endpoints_by_ids' .

Repository: NVIDIA/infra-controller

Length of output: 23312


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
import re

decom = Path("crates/switch-controller/src/decommissioning.rs").read_text()
rms = Path("crates/component-manager/src/rms.rs").read_text()
redfish = Path("crates/redfish/src/libredfish/implementation.rs").read_text()
auth = Path("crates/redfish/src/libredfish/auth.rs").read_text()

def require(label, pattern, text):
    if not re.search(pattern, text, re.S):
        raise SystemExit(f"missing invariant: {label}")
    print(f"PASS: {label}")

require(
    "completed NVOS reset enters force-restart state",
    r"SwitchFactoryResetState::Completed.*?VerifyNvosDhcpReleaseState::ForceRestarting",
    decom,
)
require(
    "NVOS verify resolves endpoint before state match",
    r"async fn handle_verify_nvos_dhcp_release.*?let endpoint = resolve_switch_endpoint\(.*?\)\.await\?.*?match verifying_state",
    decom,
)
require(
    "waiting state reads only the NVOS MAC from the resolved endpoint",
    r"WaitingForDhcpAcknowledgement.*?dhcp_suppression_acknowledged\(endpoint\.nvos_mac",
    decom,
)
require(
    "force restart uses the resolved endpoint",
    r"ForceRestarting.*?power_control\(std::slice::from_ref\(&endpoint\), PowerAction::ForceRestart\)",
    decom,
)
require(
    "BMC reset happens before suppression and transition",
    r"bmc_reset_to_defaults\(\).*?let txn = suppress_dhcp.*?VerifyDhcpRelease",
    decom,
)
require(
    "BMC factory-reset state recreates client with BMC-MAC auth",
    r"FactoryResetBmc.*?RedfishAuth::for_bmc_mac\(bmc_mac_address\)",
    decom,
)
require(
    "RMS switch node carries both credential sets",
    r"bmc_credentials.*?credentials_to_rms\(&ep\.bmc_credentials\).*?nvos_credentials.*?credentials_to_rms\(&ep\.nvos_credentials\)",
    rms,
)
require(
    "Redfish key auth reads credentials before client creation",
    r"RedfishAuth::Key\(credential_key\).*?get_credentials\(&credential_key\).*?Endpoint",
    redfish,
)
require(
    "for_bmc_mac selects BMC root credentials",
    r"RedfishAuth::Key\(CredentialKey::BmcCredentials.*?BmcCredentialType::BmcRoot",
    auth,
)
PY

echo '--- narrow endpoint row/query and existing MAC-only callers ---'
cat -n crates/api-db/src/switch.rs | sed -n '720,810p'
cat -n crates/switch-controller/src/maintenance.rs | sed -n '340,370p'

Repository: NVIDIA/infra-controller

Length of output: 283


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path

decom = Path("crates/switch-controller/src/decommissioning.rs").read_text()
rms = Path("crates/component-manager/src/rms.rs").read_text()
redfish = Path("crates/redfish/src/libredfish/implementation.rs").read_text()
auth = Path("crates/redfish/src/libredfish/auth.rs").read_text()

def section(text, start, end):
    a = text.index(start)
    b = text.index(end, a)
    return text[a:b]

def ordered(label, text, *parts):
    positions = []
    cursor = 0
    for part in parts:
        pos = text.find(part, cursor)
        if pos < 0:
            raise SystemExit(f"missing invariant: {label}: {part!r}")
        positions.append(pos)
        cursor = pos + len(part)
    print(f"PASS: {label} at offsets {positions}")

verify = section(
    decom,
    "async fn handle_verify_nvos_dhcp_release(",
    "async fn handle_factory_reset_bmc(",
)
ordered(
    "NVOS verify resolves endpoint before matching state",
    verify,
    "let endpoint = resolve_switch_endpoint(",
    "match verifying_state",
)
waiting = section(
    verify,
    "VerifyNvosDhcpReleaseState::WaitingForDhcpAcknowledgement",
    "    }\n}",
)
ordered(
    "waiting state uses only the NVOS MAC after endpoint resolution",
    waiting,
    "dhcp_suppression_acknowledged(endpoint.nvos_mac",
)
bmc = section(
    decom,
    "async fn handle_factory_reset_bmc(",
    "async fn handle_verify_dhcp_release(",
)
ordered(
    "BMC reset precedes suppression and transition",
    bmc,
    "bmc_reset_to_defaults()",
    "let txn = suppress_dhcp",
    "SwitchDecommissioningState::VerifyDhcpRelease",
)
ordered(
    "BMC retry recreates client with BMC-MAC authentication",
    bmc,
    "create_client(",
    "RedfishAuth::for_bmc_mac(bmc_mac_address)",
)

node = section(rms, "fn build_switch_node_info(", "/// Builds the host-only endpoint")
ordered(
    "RMS switch node carries BMC and NVOS credentials",
    node,
    "credentials_to_rms(&ep.bmc_credentials)",
    "credentials_to_rms(&ep.nvos_credentials)",
)
ordered(
    "Redfish key auth reads credentials before endpoint creation",
    redfish,
    "RedfishAuth::Key(credential_key)",
    "get_credentials(&credential_key)",
    "let endpoint = Endpoint",
)
ordered(
    "for_bmc_mac selects BMC root credentials",
    auth,
    "CredentialKey::BmcCredentials",
    "BmcCredentialType::BmcRoot",
)
PY

echo '--- endpoint row/query and existing MAC-only callers ---'
cat -n crates/api-db/src/switch.rs | sed -n '720,810p'
cat -n crates/switch-controller/src/maintenance.rs | sed -n '340,370p'

Repository: NVIDIA/infra-controller

Length of output: 6303


Make decommissioning resilient to credential changes after factory reset.

  • After NVOS reset completion, power_control receives the stored BMC and NVOS credentials. Provide post-reset credentials or perform the force restart before the reset.
  • If bmc_reset_to_defaults succeeds but DHCP suppression or the transition commit fails, the retry recreates the client with the same stored BMC credential. Make this retry idempotent and reset-aware.
  • Resolve the full endpoint only in ForceRestarting. Use a MAC-only lookup, or persist the MAC, while waiting for DHCP acknowledgement.
📍 Affects 1 file
  • crates/switch-controller/src/decommissioning.rs#L191-L220 (this comment)
  • crates/switch-controller/src/decommissioning.rs#L277-L286
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/switch-controller/src/decommissioning.rs` around lines 191 - 220, The
decommissioning flow must remain usable after factory-reset credentials change.
In handle_verify_nvos_dhcp_release
(crates/switch-controller/src/decommissioning.rs:191-220), defer full endpoint
resolution until ForceRestarting and use a MAC-only lookup or persisted MAC
while awaiting DHCP; ensure power_control receives post-reset credentials or
force-restarts before reset. In the retry path at
crates/switch-controller/src/decommissioning.rs:277-286, make retries after
bmc_reset_to_defaults idempotent and reset-aware so DHCP suppression or
transition-commit failures do not recreate the client with stale stored BMC
credentials.

Comment on lines +15 to +17
## DESCRIPTION

Start decommissioning a managed switch

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Document the managed-switch decommissioning contract completely.

The command references omit required preconditions, asynchronous behavior, state transitions, errors, and permanent cleanup effects. Update the CLI help source or a canonical workflow page, then regenerate these references.

  • docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-decommission.md#L15-L17: document the Ready-state and RMS-backend requirements, request persistence, asynchronous progression, terminal outcome, and failure behavior.
  • docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-delete-decommissioned.md#L16-L18: document the terminal Decommissioned requirement, irreversible deletion, removed managed records, retry behavior, and unsupported paths.

As per coding guidelines, Markdown must document interface contracts and state-machine transitions completely. As per path instructions, review Markdown for technical correctness and operator usability.

📍 Affects 2 files
  • docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-decommission.md#L15-L17 (this comment)
  • docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-delete-decommissioned.md#L16-L18
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-decommission.md`
around lines 15 - 17, Update the managed-switch decommission documentation at
docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-decommission.md:15-17
to describe Ready-state and RMS-backend prerequisites, request persistence,
asynchronous progression, terminal outcome, and failure behavior; update
docs/manuals/nico-admin-cli/commands/managed-switch/managed-switch-delete-decommissioned.md:16-18
to document the terminal Decommissioned prerequisite, irreversible deletion,
removed managed records, retry behavior, and unsupported paths, then regenerate
the CLI references from the canonical source.

Sources: Coding guidelines, Path instructions

Comment thread rest-api/flow/internal/nicoapi/grpc.go
Comment thread rest-api/flow/internal/task/componentmanager/nvswitch/nico/nico.go Outdated
@thossain-nv thossain-nv added the rest-api Add this label when an issue or PR concerns NICo REST API label Aug 6, 2026 — with ChatGPT Codex Connector
@zhaozhongn
zhaozhongn requested a review from spydaNVIDIA August 6, 2026 19:44
@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

🔐 TruffleHog Secret Scan

No secrets or credentials found!

Your code has been scanned for 700+ types of secrets and credentials. All clear! 🎉

🔗 View scan details

🕐 Last updated: 2026-08-07 02:25:28 UTC | Commit: b98027c

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rest-api Add this label when an issue or PR concerns NICo REST API

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Managed switch decommissioning

2 participants