Interface kube-ipvs0 NOARP and DOWN

[ilpozzd]

What happened?

When the cluster initializes with kubeadm running, the kube-ipvs0 interface is set to NOARP and DOWN, so services in the cluster exposed through LoadBalancer are unavailable. Manual operations to set the ARP status of this interface to ON and the state to UP result in a state change to UNKNOWN and do not affect the availability of the service.

4: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether 2a:ba:0a:f2:11:c8 brd ff:ff:ff:ff:ff:ff
    inet 10.96.0.1/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.96.0.10/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.96.193.207/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.96.86.98/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 198.168.50.100/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 10.96.33.116/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever

In this case 192.168.50.100/32 - IP address received from MetalLB to Nginx Ingress controller (Nginx defaultbackend should be reachable with this IP address)

What did you expect to happen?

Interface normally starts and all services can be reacheble from LAN.

How can we reproduce it (as minimally and precisely as possible)?

Create custer in bare metal using kubeadm

Anything else we need to know?

Linux parameters:

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

modprobe overlay
modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
sysctl --system

Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", GitTreeState:"clean", BuildDate:"2022-01-19T17:35:46Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:34:54Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

VMware vSphere

OS version

# On Linux:
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
$ uname -a
Linux test-cluster-control-plane-0 5.4.0-96-generic #109-Ubuntu SMP Wed Jan 12 16:49:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Install tools

$ kubeadm version
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", GitTreeState:"clean", BuildDate:"2022-01-19T17:34:34Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

Container runtime (CRI) and and version (if applicable)

$ crio version
crio version 1.22.0
INFO[2022-01-20 07:22:32.355996464Z] Starting CRI-O, version: 1.22.0, git: 8d4d158935929800c4300b82eb4b5a83ded400f4(dirty) 
Version:          1.22.0
GitCommit:        8d4d158935929800c4300b82eb4b5a83ded400f4
GitTreeState:     dirty
BuildDate:        1980-01-01T00:00:00Z
GoVersion:        go1.16.6
Compiler:         gc
Platform:         linux/amd64
Linkmode:         static
BuildTags:        apparmor, exclude_graphdriver_devicemapper, seccomp, selinux
SeccompEnabled:   true
AppArmorEnabled:  true

Related plugins (CNI, CSI, ...) and versions (if applicable)

System Info:
  Machine ID:                 867042b19a94457a8b6d19d2eff913ef
  System UUID:                42228a11-9f76-1a60-4444-c007c24ccd59
  Boot ID:                    6fca7a08-eddb-4821-9c6f-2b28646cd8ee
  Kernel Version:             5.4.0-96-generic
  OS Image:                   Ubuntu 20.04.3 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  cri-o://1.22.0
  Kubelet Version:            v1.23.2
  Kube-Proxy Version:         v1.23.2

[k8s-ci-robot]

@ilpozzd: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ilpozzd]

/sig network

[ilpozzd]

/sig node

[pacoxu]

/area ipvs

[cyclinder]

Do you enable the strictARP? In layer2 mode and IPVS, we should enable strictARP in config of kube-proxy, refer to https://metallb.universe.tf/installation/#preparation

[ilpozzd]

Do you enable the strictARP? In layer2 mode and IPVS, we should enable strictARP in config of kube-proxy, refer to https://metallb.universe.tf/installation/#preparation

Yes, I do

[ilpozzd]

Do you enable the strictARP? In layer2 mode and IPVS, we should enable strictARP in config of kube-proxy, refer to https://metallb.universe.tf/installation/#preparation

MetalLB works correctly, because his function ends on publishing IP address to interface. But i cant reach this IP from K8s node, got "No route to host"

[uablrek]

kube-ipvs0 should be down and noarp. This is intentional and not a bug. kube-ipvs0 only purpose is to hold addresses that should be directed to ipvs. Compare with proxy-mode=iptables where the vip addresses are on no interface at all. Unfortunately there is no easy way to do that with ipvs, if it were, it would have been used and no kube-ipvs0 interface would have been defined.

The intention is absolutely not to use proxy-arp to announce the addresses on the local segment. That leads to disaster since all nodes would do arp responses on the same addresses. Metallb in L2 mode does this in a controlled way by carefully selecting one node to answer neigbor requests (ipv4 and ipv6).