bpf: egw: delay SNAT for local client to actual egress interface

[julianwiedmann]

To let EGW traffic exit the gateway through the correct interface,
we've introduced FIB lookup-driven redirects in the to-netdev path
(#26215). This is needed for cases
where the traffic first hits one interface via the default route, but then
needs to bounce to some other interface that matches the actual egressIP.
In this approach we masquerade the packet on its first pass through
to-netdev, set the SNAT_DONE mark, and then redirect to the actual egress
interface. Due to the SNAT_DONE mark we then skip the SNAT logic in the
second pass through to-netdev.

#29379 then improved the situation for
any EGW traffic that enters the gateway from the overlay network (==
anything that's not by a pod on the gateway). We now redirect in
from-overlay, straight to the actual egress interface and masquerade the
packet there.

Now also harmonize the approach for local pods, and defer the masquerade
until the packet hits the actual egress interface. This simplifies the
overall picture. But it also allows us to raise TO_NETWORK datapath trace
events that are enriched with the packet's original source IP - this event
is raised on the second pass through to-netdev, so we need the SNAT to
happen at the same time.

Also add a comment to clarify the check to skip HostFW for SNATed traffic.

[julianwiedmann]

/test

[julianwiedmann]

/test

[julianwiedmann]

/test