NVIDIA NIC Configuration Operator

NVIDIA NIC Configuration Operator provides Kubernetes API(Custom Resource Definition) to allow FW configuration on Nvidia NICs in a coordinated manner. It deploys a configuration daemon on each of the desired nodes to configure Nvidia NICs there. NVIDIA NIC Configuration Operator uses the Maintenance Operator to prepare a node for maintenance before the actual configuration.

• Kubernetes cluster
• NVIDIA Network Operator deployed. It is recommended to deploy the DOCA-OFED driver
• Maintenance Operator deployed

NVIDIA NIC Configuration Operator can be deployed as part of the NIC Cluster Policy CRD.

The NICConfigurationTemplate CRD is used to request FW configuration for a subset of devices.

NIC Configuration Operator will select NIC devices in the cluster that match the template's selectors and apply the configuration spec to them.

If more than one template matches a single device, none will be applied and the error will be reported in all of their statuses.

For more information refer to api-reference.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
   name: connectx6-config
   namespace: network-operator
spec:
   nodeSelector:
      feature.node.kubernetes.io/network-sriov.capable: "true"
   nicSelector:
      # nicType selector is mandatory the rest are optional. Only a single type can be specified.
      nicType: 101b
      pciAddresses:
         - "0000:03:00.0"
         - “0000:04:00.0”
      serialNumbers:
         - "MT2116X09299"
      partNumbers:
         - "MCX713106AEHEA_QP1"
   resetToDefault: false # if set, template is ignored, device configuration should reset
   template:
      numVfs: 2
      linkType: Ethernet
      pciPerformanceOptimized:
         enabled: true
         maxReadRequest: 4096
      roceOptimized:
         enabled: true
         qos:
            trust: dscp
            pfc: "0,0,0,1,0,0,0,0"
            tos: 0
      gpuDirectOptimized:
         enabled: true
         env: Baremetal

• numVFs: if provided, configure SR-IOV VFs via nvconfig.
  • This is a mandatory parameter.
  • E.g: if numVFs=2 then SRIOV_EN=1 and SRIOV_NUM_OF_VFS=2.
  • If numVFs=0 then SRIOV_EN=0 and SRIOV_NUM_OF_VFS=0.
• linkType: if provided configure linkType for the NIC for all NIC ports.
  • This is a mandatory parameter.
  • E.g linkType = Infiniband then set LINK_TYPE_P1=IB and LINK_TYPE_P2=IB if second PCI function is present
• pciPerformanceOptimized: performs PCI performance optimizations. If enabled then by default the following will happen:
  • Set PCI max read request size for each PF to 4096 (note: this is a runtime config and is not persistent)
  • Users can override the runtime value via maxReadRequest
• roceOptimized: performs RoCE related optimizations. If enabled performs the following by default:
  • Nvconfig set for both ports (can be applied from PF0)
    • Conditionally applied for second port if present
      • ROCE_CC_PRIO_MASK_P1=255, ROCE_CC_PRIO_MASK_P2=255
      • CNP_DSCP_P1=48, CNP_DSCP_P2=48
      • CNP_802P_PRIO_P1=6, CNP_802P_PRIO_P2=6
  • Configure pfc (Priority Flow Control) for priority 3, set trust to dscp on each PF, set ToS (Type of Service) to 0.
    • Non-persistent (need to be applied after each boot)
    • Users can override values via trust, pfc and tos parameters
  • Can only be enabled with linkType=Ethernet
• gpuDirectOptimized: performs gpu direct optimizations. ATM only optimizations for Baremetal environment are supported. If enabled perform the following:
  • Set nvconfig ATS_ENABLED=0
  • Can only be enabled when pciPerformanceOptimized is enabled
  • Both the numeric values and their string aliases, supported by NVConfig, are allowed (e.g. REAL_TIME_CLOCK_ENABLE=False, REAL_TIME_CLOCK_ENABLE=0).
  • For per port parameters (suffix _P1, _P2) parameters with _P2 suffix are ignored if the device is single port.
• spectrumXOptimized: enables Spectrum-X specific NIC optimizations. When enabled:
  • Requires linkType=Ethernet and numVfs=1
  • Cannot be combined with roceOptimized (RoCE settings are included automatically)
  • Native NVConfig precedence is rawNvConfig > template-derived parameters > doSPCX. Validation uses DMS GET _nvconfig metadata to recognize overridden typed leaves and checks their native current/next-boot values, so intentional overrides converge after reboot
  • Requires a DMS build containing NVConfig GET mapping metadata support (DOCA changes 1508833 and 1521729) when native parameters accompany typed intent. Current and pending mappings must both be present and consistent; missing or conflicting mappings fail validation before apply, including with force
  • Breakout lanes are validated from native MODULE_SPLIT_M<module> current/next-boot arrays and translated to concrete native assignments for apply. This avoids DMS breakout GET sibling reads and indexed-template compilation failures while keeping one combined DMS apply action and one underlying mlxconfig set per target. Explicit native overrides retain precedence; conflicting typed lane assignments fail before apply. Only explicitly unfiltered native snapshots are reused; filtered snapshots trigger a complete module-array refresh
  • Can be combined with networkBay: doSPCX overrides system-profile defaults, while explicit template settings and rawNvConfig take precedence over doSPCX
  • Only supported on ConnectX-7 (nicType: 1021), ConnectX-8 (nicType: 1023), ConnectX-9 (nicType: 1025) and BlueField-3 SuperNIC (nicType: a2dc)
  • version: Required. Spectrum-X architecture version passed to the doSPCX planner
  • platformType: Required. doSPCX platform identifier defined by the supplied Blueprints profile
  • overlay: Optional, default none. Set to l3 for L3 EVPN overlay
  • multiplaneMode: Optional, default none. Options: none, swplb, hwplb
  • numberOfPlanes: Optional, default 1. Options: 1, 2, or 4
• If a configuration is not set in spec, its non-volatile configuration parameters (if any) should be set to device default.

The planner profile is selected from platformType and multiplaneMode:

RTX rejects multiplane modes; VR rejects single-plane mode. The supplied data bundle must contain the selected profile and support the requested hardware. VR profile selection alone does not provide complete VR support: target-map construction still lacks the two-NIC-per-rail layout and nic_index_in_rail, and the CRD does not yet allow eight planes for VR hardware multiplane.

Spectrum-X configuration is compiled from the doSPCX data bundle published by the dospcx-data repository. The labeled ConfigMap contains a versioned format marker and a gzip-compressed archive of the complete doSPCX data tree:

apiVersion: v1
kind: ConfigMap
metadata:
  name: doscpx-data-main
  labels:
    network.nvidia.com/operator.nic-configuration.spectrum-x-profile: ""
data:
  format: dospcx-data.tar.gz/v1
binaryData:
  dospcx-data.tar.gz: <base64-encoded-archive>

The Kubernetes API decodes binaryData before it reaches the daemon. The daemon validates and extracts the archive with the Go standard library, then replaces /opt/mellanox/doca/services/dms/doSpcx/data with the restored data/ tree. Archive paths must remain under data/; links, special files, path traversal, oversized archives, and incomplete doSPCX layouts are rejected. A failed update leaves the previous valid tree in place. Exactly one labeled doSPCX data bundle may exist cluster-wide; multiple bundles are rejected instead of selecting one based on reconciliation order, and manager-installed data is deactivated while the conflict exists. Removing the active bundle removes the data installed from it and invalidates plans cached with its digest.

Select the architecture and platform inputs in a NicConfigurationTemplate:

spectrumXOptimized:
  enabled: true
  version: "ra2.2"
  platformType: "gb300"
  overlay: "none"
  multiplaneMode: "none"
  numberOfPlanes: 1

Supported NIC types for Spectrum-X:

• ConnectX-7 (device ID 1021) -- supports none only (single-plane)
• ConnectX-8 (device ID 1023) -- supports none, swplb, and hwplb
• ConnectX-9 (device ID 1025) -- supports none, swplb, and hwplb
• BlueField-3 SuperNIC (device ID a2dc) -- supports none and swplb

doSPCX profiles can configure NICs with multiple data planes. Available modes:

spectrumx.SpectrumXManager includes the spectrumx.PlanManager interface. Before the controller starts its existing concurrent per-device NV apply, it calls PreparePlan once for the node's Spectrum-X device group with the prepare stage. It does the same with the configure stage before the existing concurrent runtime apply. Plan preparation itself never executes generated plan operations. PreparePlan parses the host-k8s plan.semantic.groups contract once and caches a homogeneous configuration plan in memory. The compiled form contains only three execution inputs: ordered breakout and post-breakout XPath operation slices for the prepare stage, and ordered runtime operation groups for the configure stage. Device targeting remains the responsibility of the configuration manager when it consumes the plan. GetPreparedPlan validates the requesting device's inputs and membership against the cache; it does not reread or reparse files for every per-device apply. During NV validation, the configuration manager separately queries the existing template-derived native parameter map and the active doSPCX XPath phase. During NV apply, it sends both inputs in one DMS action per discovered PCI function, using local port 1 consistently with validation. Each batch includes that function's supported native overrides. Breakout must match current and pending state on all device ports before post-breakout is considered. force applies the complete breakout plus post-breakout intent immediately; after the breakout barrier, with-default also resends both phases so default filling cannot undo breakout. Semantic group references are authoritative, and an omitted operation kind means set, matching DMS. Configure groups eswitch and vf-lifecycle are intentionally omitted from the compiled plan; unknown groups fail closed. Plan compilation remains execution-free.

At runtime, the configuration manager validates the final desired value of every operation group on each applicable PCI function. Repeated writes remain ordered during apply, while validation compares the last write for each path and leaf across all scope and target-class batches that apply to that function. Validation keeps different scope and target-class pairs in separate DMS commands. Generic runtime configuration is applied first and doSPCX groups are applied last in semantic order. The cc group starts doca_spcx_cc before its XPath operations; in HWPLB mode it uses the first function of each NIC because the functions share one RDMA device. Other groups are applied to every discovered function. Indexed XPath queries are issued individually until DMS preserves indexed keys in batched JSON responses.

The NIC Configuration Daemon image must contain the executable at /opt/mellanox/doca/tools/doca_spcx_cc; the STIG daemon images install the doca-spcx-cc package during their build. The deprecated NicFirmwareSource.spec.docaSpcXCCUrlSource field does not install or select the runtime executable. When migrating to doSPCX plans, remove that field from firmware sources and ensure any custom daemon image provides the binary.

NCO translates its CRD multiplane modes to the public profiles supplied by the doSPCX data bundle: none selects single-plane, swplb selects SPX_NetPlugin, and hwplb selects SPX_Multiplane.

The manager creates its command executor internally. The executable DMS planner remains part of the daemon base image and resolves its authored catalog from the native DMS location where the labeled doSPCX data ConfigMap restores it: /opt/mellanox/doca/services/dms/doSpcx/data. When PreparePlan is called, files are written to:

/var/lib/blueprints/target-maps/nco-<node>-spcx.json
/var/lib/blueprints/plans/nco-<node>-spcx-prepare/plan.json
/var/lib/blueprints/plans/nco-<node>-spcx-prepare/metadata.json
/var/lib/blueprints/plans/nco-<node>-spcx-configure/plan.json
/var/lib/blueprints/plans/nco-<node>-spcx-configure/metadata.json

NICs are sorted by function-zero BDF and assigned deterministic target-map rail IDs. NCO writes the doSPCX schema-v1 target-map contract: every pre-breakout target has an ID, function-zero BDF, hexadecimal device ID, ew role, and rail. The same pre-breakout target map is passed to both stages; during configure, doSPCX resolves the post-breakout inventory after NVConfig is active. The planner does not read interfaceNameTemplate; interface naming remains an independent NCO capability.

Each stage stores a flat metadata document containing only the planner inputs, including the platform, Spectrum-X settings, planner parameters, doSPCX data archive digest, and target-map digest. Repeated calls in the same process reuse the in-memory plan. After a process restart, PreparePlan loads and compiles the saved plan once when that metadata still matches, the target-map digest is unchanged, and the semantic operations compile under NCO's execution policy. Any input change or invalid saved artifact regenerates the plan through dms-cli. Before applying an individual Spectrum-X device, the configuration manager calls GetPreparedPlan and fails without changing the device if the stage-specific plan is missing, stale, or its target map does not contain that device.

Set BLUEPRINTS_STATE_DIR to override /var/lib/blueprints.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicConfigurationTemplate
metadata:
  name: spectrum-x-multiplane-configuration
  namespace: nvidia-network-operator
spec:
  nodeSelector:
      feature.node.kubernetes.io/network-sriov.capable: "true"
  nicSelector:
      nicType: "1023" # ConnectX-8. Use "1025" for ConnectX-9, or "a2dc" for BlueField-3 SuperNIC (hwplb not supported on BF3). ConnectX-7 (`1021`) is single-plane only.
      # partNumbers:
      #   - "MCX713106AEHEA_QP1"
  template:
      numVfs: 1
      linkType: Ethernet
      spectrumXOptimized:
          enabled: true
          version: "ra2.2"
          overlay: "none"
          multiplaneMode: "hwplb" # Hardware Plane Load Balancing, ConnectX-8, ConnectX-9 only
          numberOfPlanes: 4

The NICFirmwareSource CR represents a list of url sources with NIC FW binaries archives.

To enable the NIC FW upgrade feature, NicFirmwareStorage section should be provided in the NIC Cluster Policy. There is an option to create a new PVC or use the existing one. Firmware binaries will be provisioned by a provisioner controller which will watch for NICFirmwareSource obj and provision the binaries in a shared volume enabled by the given storage class. Node agents will need to make sure that the reference NICFirmwareSource object is fully reconciled (status.state == Success) before proceeding with firmware update.

To set up a persistent NFS storage in the cluster, the example from the CSI NFS Driver repository might be used. After deploying the NFS server and NFS CSI driver, the storage class should be deployed. The name of the storage class can then be passed to the NIC Cluster Policy.

# Change this template first as it only provides an example of configuration
apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareSource
metadata:
  name: connectx-6dx-firmware-22-44-1036
  namespace: network-operator
  finalizers:
    - configuration.net.nvidia.com/nic-configuration-operator
spec:
  # a list of firmware binaries from mlnx website if they are zipped try to unzip before placing
  binUrlSources:
    - https://www.mellanox.com/downloads/firmware/fw-ConnectX6Dx-rel-22_44_1036-MCX623106AC-CDA_Ax-UEFI-14.37.14-FlexBoot-3.7.500.signed.bin.zip
  bfbUrlSource: https://example.com/bluefield3-31.41.0.bfb
status:
  state: Success
  binaryVersions:
    22.44.1036:
    - mt_0000000436
  bfbVersions:
    a2dc: 34.41.0 # BF3 NIC FW
    a2d6: 25.21.0 # BF2 NIC FW

The NICFirmwareTemplate CRD is used to request FW validation or update from a referenced NICFirmwareSource for a subset of devices.

NIC Configuration Operator will select NIC devices in the cluster that match the template's selectors and apply the configuration spec to them.

If more than one template matches a single device, none will be applied and the error will be reported in all of their statuses.

For more information refer to api-reference.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicFirmwareTemplate
metadata:
  name: connectx6dx-config
  namespace: network-operator
spec:
  nodeSelector:
    kubernetes.io/hostname: cloud-dev-41
  nicSelector:
    nicType: "101d"
    # partNumbers:
    #   - "MCX713106AEHEA_QP1"
  template:
    nicFirmwareSourceRef: connectx6dx-firmware-22-44-1036
    updatePolicy: Update

The NICDevice CRD is created automatically by the configuration daemon and represents a specific NVIDIA NIC on a specific K8s node. The name of the device combines the node name, device type and its PCI device address (Domain:Bus:Device, with : and . replaced by -). PCI device address is used for uniqueness because serial numbers are not guaranteed unique on systems with embedded NICs that share a flashed VPD image (e.g. HGX B300).

status.node is a selectable field, so the devices on one node can be listed with kubectl get nicdevices -A --field-selector status.node=<node>. The configuration daemon uses the same selector to cache only its own node's devices.

FirmwareUpdateInProgress status condition can be used for tracking the state of the FW validation/update on a specific device. If an error occurs during FW update, it will be reflected in this field. ConfigUpdateInProgress status condition can be used for tracking the state of the FW configuration update on a specific device. If an error occurs during FW configuration update, it will be reflected in this field. Before validating runtime configuration, the configuration daemon checks every discovered network interface for carrier. Runtime validation fails only when all interfaces on the NIC report NO-CARRIER, and the failure is reflected in ConfigUpdateInProgress with reason RuntimeConfigUpdateFailed until at least one interface has carrier. This allows available planes to be configured while other planes are disconnected.

for more information refer to api-reference.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicDevice
metadata:
   name: co-node-25-101b-0000-04-00
   namespace: nic-configuration-operator
spec:
   firmware:
     nicFirmwareSourceRef: connectx6dx-firmware-22-44-1036
     updatePolicy: Update
   configuration:
      template:
         linkType: Ethernet
         numVfs: 8
         pciPerformanceOptimized:
            enabled: true
status:
   conditions:
      - reason: DeviceFirmwareConfigMatch
        type: FirmwareUpdateInProgress
        status: "False"
        message: Firmware matches the requested version
      - reason: UpdateSuccessful
        type: ConfigUpdateInProgress
        status: "False"
   firmwareVersion: 20.42.1000
   node: co-node-25
   partNumber: mcx632312a-hdat
   ports:
      - networkInterface: enp4s0f0np0
        fwctlDevice: /dev/fwctl/fwctl0
        pci: "0000:04:00.0"
        rdmaInterface: mlx5_0
      - networkInterface: enp4s0f1np1
        fwctlDevice: /dev/fwctl/fwctl1
        pci: "0000:04:00.1"
        rdmaInterface: mlx5_1
   psid: mt_0000000225
   serialNumber: mt2232t13210
   type: 101b

The NicDevice CRD is created and reconciled by the configuration daemon. The reconciliation logic scheme can be found here.

The NicInterfaceNameTemplate CRD allows you to define custom naming patterns for RDMA and network device interfaces. This is useful in multiplane and multi-rail deployments, and on nodes where different groups of NICs need stable names based on their fabric role.

The operator deploys udev rules to the host to rename network and RDMA interfaces according to the specified naming template.

Multiple templates can select the same node when their railPciAddresses select disjoint NicDevice objects. Each template has its own prefixes and zero-based indices, allowing names such as ew0, ns0, and admin0 on one node. If two templates select any port of the same NicDevice, the controller rejects the proposed assignment and preserves the existing naming specs and udev rules. See the fabric-role example.

Naming ownership is per NicDevice, including every PF represented by that object. Different PFs of one NicDevice cannot be split across templates. Generated netdev names must be unique among netdevs, and generated RDMA names must be unique among RDMA devices. The same name may be used once in each namespace to align a netdev with its RDMA device.

The template uses the following placeholders for device name construction:

• %nic_id%: The index of the NIC in the flattened list of NICs
• %plane_id%: The index of the plane of the specific NIC
• %rail_id%: The index of the rail where the given NIC belongs to

List one PCI address per NicDevice in railPciAddresses. For a multi-PF device, pfsPerNic determines how many PF naming rules are generated. The %nic_id% value is shared by all PFs of that device; use %plane_id% when each PF needs a distinct name.

apiVersion: configuration.net.nvidia.com/v1alpha1
kind: NicInterfaceNameTemplate
metadata:
  name: spectrum-x-interface-names
  namespace: nvidia-network-operator
spec:
  pfsPerNic: 2
  rdmaDevicePrefix: "rdma_%nic_id%_%plane_id%_%rail_id%"
  netDevicePrefix: "net_%nic_id%_%plane_id%_%rail_id%"
  railPciAddresses:
    - ["0000:1a:00.0", "0000:2a:00.0"]
    - ["0000:3a:00.0", "0000:4a:00.0"]

The railPciAddresses field defines the PCI address to rail mapping. The first dimension is the rail index and the second dimension is the list of PCI addresses of the NICs in that rail.

Set only netDevicePrefix to rename netdevs without renaming RDMA devices. To rename only RDMA devices, set netDevicePrefix: "" explicitly and provide rdmaDevicePrefix; the netDevicePrefix field remains required by the API schema. At least one prefix must be non-empty.

The operator generates udev rules based on the template and writes them to the host. The rules are written to two separate files:

Example generated udev rules for net devices (/etc/udev/rules.d/10-nic-net-interface-naming.rules):

# Auto-generated by nic-configuration-operator
# Do not edit manually

SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="p[0-9]*", KERNELS=="0000:1a:00.0", NAME="net_0_0_0"
SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="", KERNELS=="0000:1a:00.0", NAME="net_0_0_0"
SUBSYSTEM=="net", ACTION=="add", TEST!="phys_port_name", KERNELS=="0000:1a:00.0", NAME="net_0_0_0"
SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="p[0-9]*", KERNELS=="0000:1a:00.1", NAME="net_0_1_0"
SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="", KERNELS=="0000:1a:00.1", NAME="net_0_1_0"
SUBSYSTEM=="net", ACTION=="add", TEST!="phys_port_name", KERNELS=="0000:1a:00.1", NAME="net_0_1_0"
SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="p[0-9]*", KERNELS=="0000:3a:00.0", NAME="net_1_0_1"
SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="", KERNELS=="0000:3a:00.0", NAME="net_1_0_1"
SUBSYSTEM=="net", ACTION=="add", TEST!="phys_port_name", KERNELS=="0000:3a:00.0", NAME="net_1_0_1"
SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="p[0-9]*", KERNELS=="0000:3a:00.1", NAME="net_1_1_1"
SUBSYSTEM=="net", ACTION=="add", ATTR{phys_port_name}=="", KERNELS=="0000:3a:00.1", NAME="net_1_1_1"
SUBSYSTEM=="net", ACTION=="add", TEST!="phys_port_name", KERNELS=="0000:3a:00.1", NAME="net_1_1_1"

For each PCI function, the phys_port_name match limits naming to physical ports (p0, p1, and so on). The empty-value and TEST rules preserve compatibility when the attribute is empty or unavailable. VF and SF representors such as pf0vf0 and pf0sf0 share the PF's parent PCI address but match none of these rules, so they cannot claim the PF's configured name.

Example generated udev rules for RDMA devices (/etc/udev/rules.d/10-nic-rdma-interface-naming.rules):

# Auto-generated by nic-configuration-operator
# Do not edit manually

ACTION=="add", KERNELS=="0000:1a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_0_0"
ACTION=="add", KERNELS=="0000:1a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_0_1_0"
ACTION=="add", KERNELS=="0000:3a:00.0", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_0_1"
ACTION=="add", KERNELS=="0000:3a:00.1", SUBSYSTEM=="infiniband", RUN+="/usr/bin/rdma dev set %k name rdma_1_1_1"

To include the NIC Configuration Operator as part of network configuration workflows, strict order of operations might need to be enforced. For example, SR-IOV Network Configuration Daemon pod should start AFTER the NIC Configuration Daemon has finished. To indicate when NIC configuration is in progress to the pods that depend on it, the operator manages the nvidia.com/operator.nic-configuration.wait label, which has the value false when the requested NIC configuration has successfuly been applied, and the value true when the NIC configuration is in progress. To use this mechanism, the next pods in the pipeline can add nvidia.com/operator.nic-configuration.wait=false to their node label selectors. That way, they will automatically be evicted from the node when the NICs are being configured.

The NIC Configuration Daemon itself relies on the network.nvidia.com/operator.mofed.wait=false label to be present on the node as it requires the DOCA-OFED driver to be running for some of the configurations.

During device discovery, the daemon queries firmware version and PSID with flint. If flint cannot query a PCI function, the daemon reads the running firmware version and PSID from the function's RDMA sysfs attributes (fw_ver and board_id). Both values are required; if neither source provides complete firmware information, discovery follows the existing error or SKIP_DEVICE_ON_DISCOVERY_ERROR behavior.

PCI VPD metadata is optional during discovery. NICs with missing, unreadable, or invalid VPD still get a NicDevice CR with unavailable status.serialNumber, status.partNumber, and status.modelName fields set to empty strings. Available fields are retained, including metadata read from another PCI function of the same NIC. This applies regardless of SKIP_DEVICE_ON_DISCOVERY_ERROR. Without a VPD model name, status.superNIC is false. Select these NICs by node, NIC type, or PCI address rather than unavailable serial/part numbers. ConnectX-9 Network Bay pairing and Network Bay templates require a nonempty serial number.

Feature flags can be enabled via environment variables in the helm chart or NVIDIA Network Operator's NicClusterPolicy. Supported flags:

• FW_RESET_AFTER_CONFIG_UPDATE=true: explicitely reset the NIC's Firmware before the reboot and after updating its non-volatile configuration. Might be required on DGX servers where configuration update is not successfully applied after the warm reboot.

• SKIP_VM_CHECK=true: pass --skip_vm_check to mlxfwreset resetting NIC firmware while running on VMs.

• SKIP_DEVICE_ON_DISCOVERY_ERROR=true: skip a physical NIC during device discovery when firmware/PSID lookup or BlueField operation-mode lookup fails. When unset, these errors fail discovery so the daemon retries. Existing NicDevice CRs are preserved for skipped devices to avoid delete/recreate churn during transient per-device discovery failures.