datapath: Fix ENI egress routing table for cilium_host IP

[gandro]

On ENI, we install source-based egress routing rules that steer traffic from Cilium-mananged IPs (i.e. pods, but also the health IP, ingress IP and router IP) to the correct egress interface.

For pod IPs, this is done in the CNI plugin:

cilium/plugins/cilium-cni/interface.go

Lines 59 to 63 in 7875f6a

	if err := routingInfo.Configure(
	ipConfig.Address.IP,
	int(conf.DeviceMTU),
	conf.EgressMultiHomeIPRuleCompat,
	); err != nil {

For ingress and health IP, this is done from cilium-agent:

cilium/daemon/cmd/ipam.go

Lines 401 to 405 in ed20c8a

	if err := ingressRouting.Configure(
	result.IP,
	d.mtuConfig.GetDeviceMTU(),
	option.Config.EgressMultiHomeIPRuleCompat,
	); err != nil {

cilium/cilium-health/launch/endpoint.go

Lines 329 to 333 in e494302

	if err := routingConfig.Configure(
	healthIP,
	mtuConfig.GetDeviceMTU(),
	option.Config.EgressMultiHomeIPRuleCompat,
	); err != nil {

For the router (aka cilium_host) IP however, this was done differently. Commit f34371c added a new routing.SetupRules function that duplicated parts of the routing.RoutingInfo.Configure logic, but missed a crucial part: Namely the creation of the per-ENI routing table that the source-based egress rule points towards.

This means that if the cilium_host IP address was allocated from a different ENI than the pod, health and ingress IP addresses, that the routing table for that ENI was never created. This led to connectivity issues, in particular in combination with IPSec.

This commit addresses that issue by having the router IP use the same code path as the other IP users: Using RoutingInfo.Configure. This not only fixes the bug, but removes some code that was otherwise only used for the router IP.

There is one major difference between other users of RoutingInfo.Configure and the newly introduced use for the cilium_host IP: For the cilium_host IP, we skip the creation of the ingress rule (by passing in host=true), as otherwise the cilium_host IP would not be considered a local address of the host network namespace. This is consistent with the old SetupRules function did also not create such an ingress rule.

Long-term, it remains questionable if the setup of egress rules in ENI mode should be left to IPAM clients, as every client seems to do it slightly differently. Maybe this is better done by either the IPAM subsystem or a separate device manager.

Fixes: f34371c ("ipam: Add routes for cilium_host ENI address")

[gandro]

/ci-eks

[gandro]

I have locally tested this by installing Cilium with:

helm upgrade --install cilium ./install/kubernetes/cilium \
  --namespace kube-system \
  --set eni.enabled=true \
  --set ipam.mode=eni \
  --set egressMasqueradeInterfaces=eth0 \
  --set routingMode=native \
  --set endpointHealthChecking.enabled=false \
  --set image.override=quay.io/cilium/cilium-ci:f4b9743ac75ed66de00eb9d613ae2ae4cc85b9f9 \
  --set image.pullPolicy=IfNotPresent \
  --set=debug.enabled=true

And then adding a node to the cluster which does not have any pods scheduled on it (by scaling the node group).

Since the cilium_host is the only IP allocated on that new node (it's important to turn off endpointHealthChecking for that), we can see that before the PR, the ENI routing table was missing, where as after, the table is there as expected:

Before (running main):

root@ip-192-168-151-181:/home/cilium# ip addr show dev cilium_host
4: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether 8e:38:75:5f:f9:f6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.155.214/32 scope global cilium_host
       valid_lft forever preferred_lft forever
    inet6 fe80::8c38:75ff:fe5f:f9f6/64 scope link 
       valid_lft forever preferred_lft forever
root@ip-192-168-151-181:/home/cilium# ip rule
9:	from all fwmark 0x200/0xf00 lookup 2004
100:	from all lookup local
109:	from all fwmark 0x80/0x80 lookup main
111:	from 192.168.155.214 to 192.168.0.0/16 lookup 10
1024:	from all fwmark 0x80/0x80 lookup main
32766:	from all lookup main
32767:	from all lookup default
root@ip-192-168-151-181:/home/cilium# ip route show table 10
Error: ipv4: FIB table does not exist.
Dump terminated

After (upgrade):

root@ip-192-168-151-181:/home/cilium# ip addr show dev cilium_host
4: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default qlen 1000
    link/ether 8e:38:75:5f:f9:f6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.155.214/32 scope global cilium_host
       valid_lft forever preferred_lft forever
    inet6 fe80::8c38:75ff:fe5f:f9f6/64 scope link 
       valid_lft forever preferred_lft forever
root@ip-192-168-151-181:/home/cilium# ip rule
9:	from all fwmark 0x200/0xf00 lookup 2004
100:	from all lookup local
109:	from all fwmark 0x80/0x80 lookup main
111:	from 192.168.155.214 to 192.168.0.0/16 lookup 10
1024:	from all fwmark 0x80/0x80 lookup main
32766:	from all lookup main
32767:	from all lookup default
root@ip-192-168-151-181:/home/cilium# ip route show table 10
default via 192.168.128.1 dev eth0 proto kernel 
192.168.128.1 dev eth0 proto kernel scope link 

[gandro]

/test

[rgo3]

Net negative LOC 🚀

[borkmann]

LGTM! 🚀

[aditighag]

Appreciate the fix, and the detailed commit description. 🚀

My first thought after reading the commit description was thinking about the case(s) where users would hit such an error condition as cilium_host is one of the core interface that the agent manages.

I see that the fix is already marked for backports to older branches. 👍 Could you comment on how we could identify such issues? Was there a specific error message logged when the agent tried installing rules when the routing table wasn't created? (I suppose users could notice missing routing table(s) by running ip route commands.
On a related note, how can we prevent such cases in future for cilium managed devices? Creating a routing table in specific error conditions while setting up ENI rules is a potential option.

[gandro]

Could you comment on how we could identify such issues? Was there a specific error message logged when the agent tried installing rules when the routing table wasn't created?

The bug that was surfaced was that IPsec encrypted traffic was leaving on the wrong interface, and not on the egress interface the router IP was attached to. This lead to traffic being dropped by AWS network (which performs source IP validation). The wrong egress interface was expected, since the egress rule was defunct (without a routing table, it can't do anything). There was no log message, just broken IPsec encryption. Thus also the backport.

The main reason why we did not discover this broken egress rule earlier is that it only surfaces under the following circumstances: The router IP must be allocated from a different ENI than the pod (workload) IPs. As soon as there is at least one pod IP sharing the same ENI as the router IP, the CNI ADD call (that led to to the pod IP being allocated) would created the missing table. Because IP allocation is random, the bug basically only surfaces if one has lots of pre-allocated IPs (e.g. using min-allocate) distributed over multiple interfaces, but not many pods scheduled on the node yet.

On a related note, how can we prevent such cases in future for cilium managed devices? Creating a routing table in specific error conditions while setting up ENI rules is a potential option.

Yeah, the idea of an ENI specific device manager has been floated around. I have an open TODO on my plate to write down my thoughts on this.