Conserve PCIe bandwidth

[lukego]

PCIe bandwidth is more scarce than it used to be. The host interface has to be designed to use PCIe bandwidth efficiently. This will probably involve organizing DMA into fewer longer contiguous transfers rather than more smaller scattered ones.

How come PCIe bandwidth is more scarce? It's because Ethernet bandwidth has been increasing in powers of 10 while PCIe bandwidth in powers of 2. The basic numerical relationship has changed with the transition from 10G/40G to 25G/100G:

Ethernet bandwidth	PCIe bandwidth	PCIe-to-Ethernet ratio
10G	16G (PCIe 2.0 x4)	1.6x
40G	64G (PCIe 3.0 x8)	1.6x
25G	32G (PCIe 3.0 x4)	1.28x
50G	64G (PCIe 3.0 x8)	1.28x
100G	128G (PCIe 3.0 x16)	1.28x
200G	256G (PCIe 4.0 x16)	1.28x

In the good old days of 10G/40G the PCIe links had 60% extra capacity for overhead such as transferring DMA descriptors and for the PCIe protocols themselves. These modern times of 25G/100G are leaner and only 28% is available. This means that we must treat PCIe bandwidth as a scarce resource because any wastage is likely to actually impact operational performance.

[emmericp]

Recommended reading: R. Neugebauer, G. Antichi, J. Zazo, Y. Audvevich, S. López-Buedo, A. Moore, Understanding PCIe performance for end host networking, SIGCOMM 2018

The paper isn't available yet, it will probably appear by early July.
They've done some really interesting experiments there (I read an earlier draft last year). I'm sure the authors are happy to share a pre-print with you if you ask nicely.