netkvm: TX Completion Latency with Backend Interrupt Coalescing

[zjmletang]

Problem Statement

We've encountered latency issues in serial I/O scenarios (e.g., xcopy over SMB) when the backend device has interrupt coalescing enabled. The completion of each TX packet may be delayed by the coalescing interval (typically 1-10ms), significantly impacting throughput for workloads that send packets sequentially.

Current TX Completion Triggers

If I understand correctly, there are currently two ways to trigger TX completion processing:

1. Interrupt

The device sends an interrupt when TX completes. However, with backend interrupt coalescing, this may be delayed by the coalescing interval.

2. Piggyback in DoPendingTasks

When sending a new packet, the driver checks for completions opportunistically. This works well for continuous traffic but doesn't help when the next packet must wait for the previous one to complete.

The Gap

Serial I/O Timeline:
Send Pkt1 → [waiting for completion...] → Backend coalescing → Interrupt → Complete → Send Pkt2
                     ↑
            This wait seems unavoidable with current mechanisms

For serial workloads, neither mechanism can reduce the waiting time - interrupt is delayed by coalescing, and piggyback has nothing to piggyback on.

Question for Discussion

Is there interest in adding a mechanism to actively poll for TX completions independent of interrupts?

One rough idea would be a configurable timer-based polling, but this is just an initial thought - there may be better approaches. I'm also looking into whether Linux virtio-net has any reference implementation for this scenario.

Looking forward to hearing thoughts from the community.

[ybendito]

@zjmletang For me the value of the coalescing > 1 ms seem completely weird, I saw typical value of several microseconds and also with such values the TCP ack (which is not a duplicated ack) tends to stop the coalescing. The resolution of the system timer is not so predictable, probably it is not possible to achieve the timer resolution less than 1 ms. In case of several queues - which on(es) should we poll? IMO if we have a problem with host adapter configuration - we should solve it.

[zjmletang]

@ybendito Thanks for the insight. I agree that a driver-side workaround isn't the right architectural path for upstream.

I brought this up as a temporary workaround because backend changes—especially in hardware-backed scenarios—often have a very slow release cycle. Historically, we kept the coalescing at 1ms because we once encountered issues on older Linux guests when setting it to ~100μs: the TX descriptors were reclaimed by subsequent transmission flows before the interrupt fired. This resulted in interrupts with nothing to process, and the OS eventually disabled the interrupt line, leading to network outages. We've been cautious about lowering it since then.

Given your feedback, I realize that even this driver-side workaround is likely not feasible or the right approach. I’ll take this back to our team to re-evaluate our backend tuning instead. Thanks again for the perspective!