Investigate dynamic NIC RSS indirection-table tuning for runtime flow rebalancing

[psaab]

Problem

iperf3 -P N at low N (12, 32) on shared_exact classes shows high per-flow CoV (49–65 %) because NIC RSS hashes the N source ports unevenly across the 6 worker-bound RX queues. The 12-port test in the diagnostic produced a 0/2/2/2/3/3 distribution; one worker sat idle.

Project memory has `#840 reverted: tuning the NIC RSS indirection table at runtime can't fix cross-binding skew with long-lived flows`. The reason: RSS hash is computed at packet ingest based on 5-tuple, and changing the indirection table afterward only redirects FUTURE hash buckets — existing TCP flows' source-port hashes still map to the same indirection slot, which now points to a different queue, but the flow's MID-STREAM packets get redirected to a worker that has no session state for them.

Re-evaluation possibilities

Despite #840's revert, runtime RSS tuning might still help if combined with session-state migration:

1. Per-flow XDP-side steering: keep RSS as-is at the NIC, but an ingress XDP program reads a per-flow override table (set by a userspace controller that detects imbalance) and redirects matching packets to a different AF_XDP queue. The override table is keyed on 5-tuple hash; entries are added when a worker reports oversubscription and removed when the flow ends. This is essentially a hot-path `bpf_redirect_map` keyed on a per-flow LRU map.

2. Toeplitz-key rotation with grace period: rotate the NIC's Toeplitz hash key on a fixed schedule (e.g. every 60s) to redistribute long-lived flows. Existing flows continue to be served on their pre-rotation queue (via session state) while new flows hash via the new key. Trades one-time disruption for ongoing fairness.

3. Per-binding queue hints in Cross-binding fairness via per-flow XDP_REDIRECT (alternative to RSS table tuning) #899-style design: if Cross-binding fairness via per-flow XDP_REDIRECT (alternative to RSS table tuning) #899 (cross-binding redirect) lands, the NIC RSS table becomes less load-bearing — flows can be re-steered post-RSS.

Acceptance criteria

• A measurement experiment showing whether option 1 (per-flow XDP-side override) clears the iperf-b P=12 CoV gate.
• If yes: integrate with a userspace controller that detects per-worker imbalance and writes overrides.
• No regression on session-state correctness (mid-stream packets must reach the worker holding the session, not be dropped or routed to slow path).

References

• Slice D v2: RSS rebalance with xpf-userspace-dp per-binding RX stats (replaces ethtool -S signal) #840 (reverted runtime RSS tuning attempt).
• Cross-binding fairness via per-flow XDP_REDIRECT (alternative to RSS table tuning) #899 (cross-binding redirect — companion).
• Diagnostic: `docs/pr/917-mqfq-phase4/diagnostic.md`.

[psaab]

Closing per hostile review (Codex + Gemini consensus).

The proposed solutions in this issue are non-viable or duplicative:

1. Per-flow XDP-side override: identical to option 1 in Re-evaluate #899 cross-binding flow re-steering — RSS-degenerate case path #937. No new content.
2. Toeplitz-key rotation with grace period: doesn't address the LONG-LIVED-flow case which is the dominant cause of imbalance in iperf-c P=12. Existing flows continue hashing to their pre-rotation queues; rotation only helps NEW flows. Operationally complex (in-flight packets during rotation can land on a worker without session state).
3. Companion to Cross-binding fairness via per-flow XDP_REDIRECT (alternative to RSS table tuning) #899: the only viable mechanism (XDP_REDIRECT to per-worker queue at ingress) is already covered by Re-evaluate #899 cross-binding flow re-steering — RSS-degenerate case path #937 verbatim.

The premise of #840 (which this issue tries to revisit) was correctly rejected: NIC RSS table tuning at runtime cannot fix cross-binding skew with long-lived flows. That conclusion still holds; nothing in this issue changes the analysis.

All meaningful work in this space lives in #937. Refer there for the architectural re-evaluation of cross-binding redirect.

(Closing per agreed Codex + Gemini synthesis. If a follow-up surfaces a new mechanism not covered by #937 — e.g., a hardware-level RSS reprogramming primitive that DOES preserve session affinity — file a new issue then rather than reopening this one.)