Avoid IoVecBuffer allocations in the TX network data path

[bchalios]

Description

When the guest wants to transmit an ethernet frame over a virtio-net device it places the frame somewhere in guest memory and sends us an interrupt. A single frame might be placed in multiple buffers in memory.
So, our virtio-net implementation, does scatter-gather writes to the backing TAP device (call writev once with mutliple buffers, instead of write many times) to increase throughput on the TX path. We do that with
the help of a type we call IoVecBuffer. This is, essentially an abstraction layer on top of a buffer scattered in guest memory.

The current implementation creates a new IoVecBuffer item for every ethernet frame we forward from the guest into the TAP device. This implies memory allocations across a hot-path that we would like to avoid.

We can avoid the multiple allocations if we change the implementation of IoVecBuffer to allow for re-using, i.e. we create the IoVecBuffer once during device initialization and then re-use it for all TX frames.

Solution

• Make the change in the API of IoVecBuffer and integrate it with virtio-net device.
• Ensure everything is working as expected (all network-related tests keep passing).
• For bonus points: check for potential performance benefits on the TX path.

[JackThomson2]

Hey, I'll take a look at this! I'll keep you posted on how I get on

I have my initial branch here: https://github.com/JackThomson2/firecracker/tree/iovec_reuse