Supporting Nvidia Cx7 400GB

[SteveSMC]

Hi iPerf Team,

Was wondering if iPerf supports Nvidia CX-7 400GB NICs? I'm only able to get close to 200GB bandwidth using UDP and P > 10. Please advise what's the max bandwidth iPerf3 supports.

[davidBar-On]

(I assume you mean 400Gb or 400bps, as 400MB means 400 mega bytes)
As far as I know there is no such limit. The maximum throughput depends on several parameters, such as: sending and receiving computers performance, network buffers size, network performance (having 400Gbps NICs does not mean the network supports this throughput), packets size, etc.

I suggest to first check the CPU load of the sending computer (the client computer if you didn't use --reverse or --bidir) and on the receiving computer. If the CPU load is close to 100% then the CPU is the limit.

Also, you can run the sender with no limit to the bandwidth (not use the -b option). The sender will then send in the maximum throughput allowed by the sending computer resources. This will show if the computer resources are the limit. (Note that with UDP, the real throughput is what is received on the receiver size, not the sending rate, as packets may be lost in the way.)

To increase performance try using some iperf3 options : -l to send larger packets, -Z and --skip-copy (and -Z) to improve CPU load.

Other actions may be increasing the network buffers, etc., but this is beyond iperf3 scope.

[bmah888]

So in addition to the points above...

We've discovered with 100GE and faster network adapters, achieving high through can also depend on the CPU and I/O topology of the host. Pinning the iperf3 process and thread(s) to CPU cores located "close" to the NIC can make a significant difference. iperf3's -A flag (to set CPU affinity) isn't quite expressive enough to handle this well, but (at least on Linux) we've had good luck with using numactl -N for this. Note that this is a consideration for "actual" high-performance data transfer applications as well.

(There is no intrinsic limit within iperf3 that would limit the speed.)