iptables: Fix wrong use of podCIDR in cluster node NAT exclusion

By default, in the iptables-based masquerading mode, Cilium will only
masquerade traffic coming from the local pod CIDR (`allocRange` in
`installMasqueradeRules`). However, many IPAM modes such as ENI or
multi-pool IPAM do not have a single pod CIDR. Instead, those modes rely
on the `egress-masquerade-interfaces` setting, which masquerades all
traffic if it leaves one of the `egress-masquerade-interfaces` devices.

Therefore, the "exclude traffic to cluster nodes from masquerade"
`CILIUM_POST_nat` rule should also respect the
`egress-masquerade-interfaces` setting and not masquerade traffic
regardless of the value of `allocRange` (which will not be valid in
settings such as ENI mode).

This likely has not manifested in ENI mode as an issue, because in ENI
mode we derive the native routing CIDR (`snatDstExclusionCIDR` in
`installMasqueradeRules`) from the EC2 VPC CIDR, which usually contains
the node IPs too. However, we should not rely on that, since we are
adding additional non-podCIDR based IPAM modes such as multi-pool where
this will not be true.

Related: #22273

Signed-off-by: Sebastian Wicki <sebastian@isovalent.com>

pkg/datapath/iptables/iptables.go

@@ -1193,13 +1193,26 @@ func (m *IptablesManager) installMasqueradeRules(prog iptablesInterface, ifName,
 			return err
 		}
 
-		if err := prog.runProg([]string{
+		progArgs := []string{
 			"-t", "nat",
 			"-A", ciliumPostNatChain,
-			"-s", allocRange,
+		}
+
+		// If EgressMasqueradeInterfaces is set, we need to mirror base condition
+		// of the "cilium masquerade non-cluster" rule below, as the allocRange might
+		// not be valid in such setups (e.g. in ENI mode).
+		if option.Config.EgressMasqueradeInterfaces != "" {
+			progArgs = append(progArgs, "-o", option.Config.EgressMasqueradeInterfaces)
+		} else {
+			progArgs = append(progArgs, "-s", allocRange)
+		}
+
+		progArgs = append(progArgs,
 			"-m", "set", "--match-set", prog.getIpset(), "dst",
 			"-m", "comment", "--comment", "exclude traffic to cluster nodes from masquerade",
-			"-j", "ACCEPT"}); err != nil {
+			"-j", "ACCEPT",
+		)
+		if err := prog.runProg(progArgs); err != nil {
 			return err
 		}
 	}