In the current Catnip LibOs, once we've decided to send a packet, we always allocate a new DPDK MBuf to hold the headers. Then depending upon the size of the data in the body, and whether the body data is already contained in an MBuf or is in a heap-allocated buffer, we (a) copy the body data into the same MBuf as the headers, (b) use the existing body MBuf as is, or (c) allocate a new MBuf to hold the body data (this latter case is only if we have a heap-allocated body buffer to start with). The decision between (a) and (b) generally comes down to whether it is quicker to copy the data versus directing the NIC's DMA engine to pull from two regions instead of one. For a small enough amount of data, the copy can actually be quicker (cache considerations may also come into play).

We're missing another option that could potentially be even faster. If we have a body MBuf (i.e., DPDK-allocated) to send, and there is "headroom" in that MBuf to hold the packet headers, we could just prepend the headers into the empty space before the body data in the MBuf and send it. This option would require no MBuf allocations and require the NIC's DMA engine to pull from just a single region. It's a win/win.

Note that DPDK already supports the option to allocate MBufs that contain a configurable amount of "headroom" (which is extra empty space in the MBuf that comes before the region given to the user to write their data into).