cilium: encryption, fixes for ENI & Azure mode with shared podIPs and networkIPs

[jrfastab]

Subnet mode came with some baked in assumptions that are not met in all ENI use cases. First we assumed the networkIPs would not overlap with the podIPs. This was important because we use the same routing table for both encrypt and decrypt with the above assumption this works because encrypted packets and decrypted packets will match different routes and never collide. But, once this assumption is broke depending on how subnets and IPs are chosen its possible they collide. The result is a encrypted packet may be routed as if it has already been decrypted or vice versa. We've seen some cases where the overlap happens in EKS and AKS.

Next we made an assumption that cilium_host interface would always understand how to handle packets. In ENI mode with endpoint routes enabled we optimized cilium_host recently so it would not handle ingress packets. However, we missed the subnet encryption case combination of features. Fix it here by also enabling subnet encryption to work with endpoint routes. We took this opportunity to make a small optimization and cut out an extra pass through bpf_network as it was no longer needed with above fixes.

Finally, feature detection incorrectly detected the ingress interface in some cases.

Quick note, I broke the patches down into small chunks. Other than the refactor patches they are hopefully small and straightforward. Patch List

patches 1- 2 refactor code so we can push optional flag down to policy rules and program fwd policy directly
patch3: set optional flag on all fwd policies, we don't use fwd polices for policy (to drop packets not encrypted for example) because we can enforce these policies in the BPF programs and drop any non-compliant packets early in the stack at XDP or TC layers. Without this endpoint routes + one CIDR for network and podIPs can cause a drop.
patch4: subnetEncryption was always enabling nodeEncrypt even if it wasn't configured this is wrong.
patch5: ENI needs to use linkAddr for IPSec tunnel IPs
patch6: ENI, redirecting to cilium_host breaks endpoint routes
patch7: simplify stack to avoid extra loop by having xfrm stack clear mark bits.
patch8: remove mark from fwd rule to make it more permissive again we don't use it for policy
patch9: clear mark from xfrm states so we can skip extra pass through eth0
patch10: ENI auto iface detection doesn't in my deployment it picks the wrong interface, this makes it so we use the user supplied interface if its given.

Couple pictures to help with understanding ingress datapath.

Picture of possible failing case if subnet IPs overlap and then collide with interface IPs.

Picture of fixed datapath to support overlapping subnet IPs.

[jrfastab]

I'll pop patch1 off in a bit it fixes a build env issue on my system

[jrfastab]

@joestringer We need to do this in two steps. First get the extra option in the netlink code. Here is the PR cilium/netlink#1