Double handoff for small single-packet RPCs

[breakertt]

Dear John,

I noticed that for small single-packet messages, when unloaded, homa_copy_to_user gets called roughly 1.5x per RPC.

After a rough check, I realized that the handoff in homa_rpc_alloc_server triggers the syscall thread and sets the RPC_PKTS_READY bit, but does not add the skb into the msgin.packets, which caused homa_copy_to_user exit and clear the RPC_PKTS_READY bit before the skb is added. This caused softirq call handoff again. This happens roughly 50% of the time for small (<MTU) unloaded RPCs.

I tried to elimitate thiswith two ways below

1. Add packet to msgin.packets in alloc_server
2. Check skb_queue_len > 0 before taking an RPC off ready_rpcs in homa_wait_shared, then look a while

Both solutions improved unloaded latency for 0.3 us but led to 10-30% loaded throughput decrease, which is not a good trade-off imo. Therefore, I wonder if you are aware of this problem already as intended design, or if you have a better fix that can come up with? Thanks.

Cheers,
TIanyi