NodePort fails when using IP attached to interface without HW address

[donch]

Bug report

General Information

• Cilium version : 1.8.0
• Kernel version : 5.3.0-59-generic
• Orchestration system version in use : v1.15.9

How to reproduce the issue

When deploying Cilium, network iface auto-discovery doesn't work with our networking design. We need to explicitly define --global.devices=eth0 to have Cilium running, but in this configuration, we are not able to establish a connection to a NodePort service.

I can provide some details on our network stack :

• IPs to use are attached to the loopback, and are exported using Exabgp. The physical device is 'eth0'

default proto exabgp src 10.189.20.54 metric 500
	nexthop via 10.129.119.2 dev eth0 weight 1
	nexthop via 10.129.119.1 dev eth0 weight 1

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.189.20.54/32 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 20:47:47:80:a8:d4 brd ff:ff:ff:ff:ff:ff
    inet 10.129.119.3/25 brd 10.129.119.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::2247:47ff:fe80:a8d4/64 scope link
       valid_lft forever preferred_lft forever

With this setup, Cilium's startup script is not able to detect eth0 as network interface

level=warning msg="+ set -o pipefail" subsys=datapath-loader
level=warning msg="++ command -v cilium-map-migrate" subsys=datapath-loader
level=warning msg="+ [[ ! -n /usr/bin/cilium-map-migrate ]]" subsys=datapath-loader
level=warning msg="+ rm /var/run/cilium/state/encap.state" subsys=datapath-loader
level=warning msg="+ true" subsys=datapath-loader
level=warning msg="+ DIR=/run/cilium/state/globals" subsys=datapath-loader
level=warning msg="+ case \"${MODE}\" in" subsys=datapath-loader
level=warning msg="+ HOST_DEV1=cilium_host" subsys=datapath-loader
level=warning msg="+ HOST_DEV2=cilium_net" subsys=datapath-loader
level=warning msg="+ setup_veth_pair cilium_host cilium_net" subsys=datapath-loader
level=warning msg="+ local -r NAME1=cilium_host" subsys=datapath-loader
level=warning msg="+ local -r NAME2=cilium_net" subsys=datapath-loader
level=warning msg="++ ip link show cilium_host type veth" subsys=datapath-loader
level=warning msg="++ cut -d ' ' -f 2" subsys=datapath-loader
level=warning msg="+ '[' cilium_host@cilium_net: '!=' cilium_host@cilium_net: ']'" subsys=datapath-loader
level=warning msg="+ setup_dev cilium_host" subsys=datapath-loader
level=warning msg="+ local -r NAME=cilium_host" subsys=datapath-loader
level=warning msg="+ ip link set cilium_host up" subsys=datapath-loader
level=warning msg="+ '[' '<nil>' '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ '[' 10.174.66.75 '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="+ setup_dev cilium_net" subsys=datapath-loader
level=warning msg="+ local -r NAME=cilium_net" subsys=datapath-loader
level=warning msg="+ ip link set cilium_net up" subsys=datapath-loader
level=warning msg="+ '[' '<nil>' '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ '[' 10.174.66.75 '!=' '<nil>' ']'" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="+ echo 1" subsys=datapath-loader
level=warning msg="+ echo 0" subsys=datapath-loader
level=warning msg="+ ip link set cilium_host arp off" subsys=datapath-loader
level=warning msg="+ ip link set cilium_net arp off" subsys=datapath-loader
level=warning msg="+ ip link set cilium_host mtu 1500" subsys=datapath-loader
level=warning msg="+ ip link set cilium_net mtu 1500" subsys=datapath-loader
level=warning msg="+ case \"${MODE}\" in" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*CILIUM_NET_MAC.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="++ ip link show cilium_net" subsys=datapath-loader
level=warning msg="++ grep ether" subsys=datapath-loader
level=warning msg="++ awk '{print $2}'" subsys=datapath-loader
level=warning msg="+ CILIUM_NET_MAC=ca:40:9c:dc:a2:49" subsys=datapath-loader
level=warning msg="++ mac2array ca:40:9c:dc:a2:49" subsys=datapath-loader
level=warning msg="++ echo '{0xca,0x40,0x9c,0xdc,0xa2,0x49}'" subsys=datapath-loader
level=warning msg="+ CILIUM_NET_MAC='{0xca,0x40,0x9c,0xdc,0xa2,0x49}'" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*CILIUM_NET_MAC.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="+ echo '#ifndef CILIUM_NET_MAC'" subsys=datapath-loader
level=warning msg="+ echo '#define CILIUM_NET_MAC { .addr = {0xca,0x40,0x9c,0xdc,0xa2,0x49}}'" subsys=datapath-loader
level=warning msg="+ echo '#endif /* CILIUM_NET_MAC */'" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*HOST_IFINDEX.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="++ cat /sys/class/net/cilium_net/ifindex" subsys=datapath-loader
level=warning msg="+ HOST_IDX=8" subsys=datapath-loader
level=warning msg="+ echo '#define HOST_IFINDEX 8'" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*HOST_IFINDEX_MAC.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="++ ip link show cilium_host" subsys=datapath-loader
level=warning msg="++ grep ether" subsys=datapath-loader
level=warning msg="++ awk '{print $2}'" subsys=datapath-loader
level=warning msg="+ HOST_MAC=ca:4f:b3:98:ea:90" subsys=datapath-loader
level=warning msg="++ mac2array ca:4f:b3:98:ea:90" subsys=datapath-loader
level=warning msg="++ echo '{0xca,0x4f,0xb3,0x98,0xea,0x90}'" subsys=datapath-loader
level=warning msg="+ HOST_MAC='{0xca,0x4f,0xb3,0x98,0xea,0x90}'" subsys=datapath-loader
level=warning msg="+ echo '#define HOST_IFINDEX_MAC { .addr = {0xca,0x4f,0xb3,0x98,0xea,0x90}}'" subsys=datapath-loader
level=warning msg="+ sed -i '/^#.*CILIUM_IFINDEX.*$/d' /var/run/cilium/state/globals/node_config.h" subsys=datapath-loader
level=warning msg="++ cat /sys/class/net/cilium_host/ifindex" subsys=datapath-loader
level=warning msg="+ CILIUM_IDX=9" subsys=datapath-loader
level=warning msg="+ echo '#define CILIUM_IFINDEX 9'" subsys=datapath-loader
level=warning msg="++ cat /proc/sys/net/ipv4/ip_local_port_range" subsys=datapath-loader
level=warning msg="++ awk '{print $1}'" subsys=datapath-loader
level=warning msg="+ CILIUM_EPHEMERAL_MIN=32768" subsys=datapath-loader
level=warning msg="+ echo '#define EPHEMERAL_MIN 32768'" subsys=datapath-loader
level=warning msg="+ '[' true = true ']'" subsys=datapath-loader
level=warning msg="+ MAC_BY_IFINDEX_MACRO='#define NATIVE_DEV_MAC_BY_IFINDEX(IFINDEX) ({ \\" subsys=datapath-loader
level=warning msg="\tunion macaddr __mac = {.addr = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}; \\" subsys=datapath-loader
level=warning msg="\tswitch (IFINDEX) { \\\\\\n'" subsys=datapath-loader
level=warning msg="+ MAC_BY_IFINDEX_MACRO_END='\t} \\" subsys=datapath-loader
level=warning msg="\t__mac; })'" subsys=datapath-loader
level=warning msg="+ for NATIVE_DEV in ${NATIVE_DEVS//;/ }" subsys=datapath-loader
level=warning msg="++ cat /sys/class/net/lo/ifindex" subsys=datapath-loader
level=warning msg="+ IDX=1" subsys=datapath-loader
level=warning msg="++ ip link show lo" subsys=datapath-loader
level=warning msg="++ grep ether" subsys=datapath-loader
level=warning msg="++ awk '{print $2}'" subsys=datapath-loader
level=warning msg="+ MAC=" subsys=datapath-loader
level=error msg="Error while initializing daemon" error="exit status 1" subsys=daemon
level=fatal msg="Error while creating daemon" error="exit status 1" subsys=daemon

If i specify the flag --global.devices=eth0, and i a create a NodePort service, i'm not able to establish a connection to the IP specified on the 'lo' iface from outside of the cluster.

35   10.129.133.3:31619     NodePort       1 => 10.174.65.146:80
                                           2 => 10.174.66.115:80
36   0.0.0.0:31619          NodePort       1 => 10.174.65.146:80
                                           2 => 10.174.66.115:80

Establishing a connection to 10.189.20.54:31619 fail.

[pchaigno]

Note: #12321 should fix the above error and allow the agent to start, but it won't fix the general issue reported here. In particular:

If i specify the flag --global.devices=eth0, and i a create a NodePort service, i'm not able to establish a connection to the IP specified on the 'lo' iface from outside of the cluster.

[pchaigno]

I've updated the title to clarify that this is not limited to this particular configuration with the loopback device. Other users have encountered the same issue with TUN devices or with the WireGuard interface.

[mmack]

I'll watching here, let me know if i can test anything.