perf: O(1) established-flow NAT forward path via per-subscriber 5-tuple hash

[w180112]

Summary

• Outbound packets had no forward index: every packet of an established flow recomputed its initial candidate port and re-walked the candidate sequence until the reverse key matched. A flow that skipped N conflicted ports at creation re-paid N reverse-hash lookups on every packet for its whole lifetime — and under CGNAT port pressure (hundreds of devices sharing one public IP) collisions are common, so the per-packet tax grows with exactly the load that matters.
• Add a second per-subscriber cuckoo hash nat_forward_hash: key = 5-tuple (src_ip, dst_ip, src_port, dst_port) (protocol-blind, matching the existing nat_entry_same_flow() semantics), value = the same addr_table slot idx the reverse hash stores. Same RW_CONCURRENCY_LF + MULTI_WRITER_ADD flags and the same shared QSBR RCU in DQ mode, but with a NULL data callback — pool-slot reclaim stays owned solely by the reverse hash's defer queue (verified against DPDK source that a NULL callback is safe on every reclaim path).
• Per-packet path becomes: one lock-free forward lookup, hit → refresh + return nat_port. Established flows never touch the candidate-port walk again; a forward miss means a genuinely new flow (or a transient GC race). With step 1, both directions are now exactly one hash lookup per packet.
• Dual-key lifecycle invariant, mirrored: inserts reverse-first (a failed forward add unpublishes via the reverse key, whose dq owns the slot — it may already be visible to readers, so it must not go straight back to the free ring), deletions forward-first (by the time the reverse delete schedules slot reclaim, no forward key references it). All three delete sites (reverse expiry-as-miss, learning eviction, GC scan) route through one helper nat_entry_del_keys(); the insert lock's double-check now covers both hashes. Forward-add ENOSPC fails the flow outright — the forward key contains no port, so the next candidate port could not succeed either.
• Fixes a step-1 latent corruption found by the new tests: nat_table_reset() refilled the free ring while deferred frees could still be pending in the RCU defer queues (visible as HASH: RCU reclaim all resources failed). A stale deferred free firing after the refill enqueued a duplicate slot index — two flows sharing one entry — and the same staleness corrupts the hashes' internal key-slot allocators. Reset now drains both defer queues first; the drain terminates because data lcores report quiescent every poll iteration (idle included) and the queues are empty at first init.
• Dead code removed: nat_entry_matches_key() (unused since step 1) plus four stale doc blocks still describing pre-step-1 signatures.
• Unit tests (add-only): test_evict_clears_forward_key (a leaked forward key would let a returning flow believe it owns a port that inbound now routes to someone else) and test_gc_clears_forward_key (a leaked forward key would fast-path into a reclaimed slot and skip restoring reverse reachability — inbound blackhole). The nat_env_reset helper now reports reader-0 quiescent before resetting so the new drain loop can complete; no existing test case modified.

This is step 2 of 3 of 項目一 of the datapath performance-improvement plan (NAT rte_hash rework; step 1: #73). Step 3 (expire_at SoA split + write-coalescing, GC forced minimum rate under sustained load, fill-rate/ENOSPC/GC stats) will come as a separate PR from a fresh branch.

Test plan

• make — production build clean, no warnings/errors
• make test — 1158/1158 unit tests pass (1148 existing + 10 new assertions; no existing case modified), controller/etcd integration tests pass
• e2e_test/run_e2e_test.sh — 73/73 pass, 0 fail, 0 skip

🤖 Generated with Claude Code