From 3ab25cab9cbb67afb2e996fa2a9e95832ce74a88 Mon Sep 17 00:00:00 2001 From: dfitzmau Date: Fri, 3 Oct 2025 12:38:55 +0100 Subject: [PATCH] OSDOCS-16309: Updated the IPoIB docs by including Bond Over Infiniband details --- ...reating-infiniband-interface-on-nodes.adoc | 43 +++++++++++++------ 1 file changed, 29 insertions(+), 14 deletions(-) diff --git a/modules/virt-creating-infiniband-interface-on-nodes.adoc b/modules/virt-creating-infiniband-interface-on-nodes.adoc index bf9e427e1043..40d15ec64b32 100644 --- a/modules/virt-creating-infiniband-interface-on-nodes.adoc +++ b/modules/virt-creating-infiniband-interface-on-nodes.adoc @@ -6,7 +6,9 @@ [id="virt-creating-infiniband-interface-on-nodes_{context}"] = Creating an IP over InfiniBand interface on nodes -On the {product-title} web console, you can install a Red{nbsp}Hat certified third-party Operator, such as the NVIDIA Network Operator, that supports InfiniBand (IPoIB) mode. Typically, you would use the third-party Operator with other vendor infrastructure to manage resources in an {product-title} cluster. To create an IPoIB interface on nodes in your cluster, you must define an InfiniBand (IPoIB) interface in a `NodeNetworkConfigurationPolicy` (NNCP) manifest file. +On the {product-title} web console, you can install a Red{nbsp}Hat certified third-party Operator, such as the NVIDIA Network Operator, that supports IP over InfiniBand (IPoIB) mode. Typically, you would use the third-party Operator with other vendor infrastructure to manage resources in an {product-title} cluster. To create an IPoIB interface on nodes in your cluster, you must define an InfiniBand (IPoIB) interface in a `NodeNetworkConfigurationPolicy` (NNCP) manifest file. + +If you need to attach IPoIB to a bond interface, only the `active-backup` mode supports this configuration. [IMPORTANT] ==== @@ -24,7 +26,6 @@ For more information about the NVIDIA Operator, see link:https://docs.nvidia.com . Create or edit a `NodeNetworkConfigurationPolicy` (NNCP) manifest file, and then specify an IPoIB interface in the file. + - [source,yaml] ---- apiVersion: nmstate.io/v1 @@ -36,8 +37,8 @@ spec: interfaces: - description: "" infiniband: - mode: datagram <1> - pkey: "0xffff" <2> + mode: datagram + pkey: "0xffff" ipv4: address: - ip: 100.125.3.4 @@ -48,21 +49,35 @@ spec: enabled: false name: ibp27s0 state: up - identifier: mac-address <3> - mac-address: 20:00:55:04:01:FE:80:00:00:00:00:00:00:00:02:C9:02:00:23:13:92 <4> - type: infiniband <5> + identifier: mac-address + mac-address: 20:00:55:04:01:FE:80:00:00:00:00:00:00:00:02:C9:02:00:23:13:92 + type: infiniband # ... ---- -<1> `datagram` is the default mode for an IPoIB interface, and this mode improves optimizes performance and latency. `connected` mode is a supported mode but consider only using this mode when you need to adjust the maximum transmission unit (MTU) value to improve node connectivity with surrounding network devices. -<2> Supports a string or an integer value. The parameter defines the protection key, or _P-key_, for the interface for the purposes of authentication and encrypted communications with a third-party vendor, such as NVIDIA. Values `None` and `0xffff` indicate the protection key for the base interface in an InfiniBand system. -<3> Supported values include `name`, the default value, and `mac-address`. The `name` value applies a configuration to an interface that holds a specified interface name. -<4> Holds the MAC address of an interface. For an IP-over-InfiniBand (IPoIB) interface, the address is a 20-byte string. -<5> Sets the type of interface to `infiniband`. ++ +-- +where: + +``:: `datagram` is the default mode for an IPoIB interface. This mode provides improved CPU performance and low-latency capabilitities for pod-to-pod communication. `connected` mode is a supported mode but consider only using this mode when you need to adjust the maximum transmission unit (MTU) value to improve node connectivity with surrounding network devices. + +``:: Supports a string or an integer value. The parameter defines the protection key, or _P-key_, for the interface for the purposes of authentication and encrypted communications with a third-party vendor, such as NVIDIA. Values `None` and `0xffff` indicate the protection key for the base interface in an InfiniBand system. + +``:: Supported values include `name`, the default value, and `mac-address`. The `name` value applies a configuration to an interface that holds a specified interface name. + +``:: Holds the MAC address of an interface. For an IP-over-InfiniBand (IPoIB) interface, the address is a 20-byte string. + +``:: Sets the type of interface to `infiniband`. +-- . Apply the NNCP configuration to each node in your cluster by running the following command. The Kubernetes NMState Operator can then create an IPoIB interface on each node. + [source,yaml] ---- -$ oc apply -f <1> +$ oc apply -f ---- -<1> Replace `` with the name of your NNCP file. ++ +-- +where: + +``:: Replace `` with the name of your NNCP file. +--