kube-vip requires super-admin.conf with Kubernetes 1.29

[louhisuo]

Describe the bug
Setting up a new Kubernetes 1.29.0 cluster with kubeadm fails when kube-vip is used. Lima-VM with Ubuntu 22.04 LTS image was used as a machine.

To Reproduce
Steps to reproduce the behavior:

1. Start a machine (virtual machine) with all Kubernetes prerequisites satisfied for Kubernetes 1.29.0

2. Within a control plane machine generate kube-vip static pod

export KVVERSION=v0.6.4
export INTERFACE=lima0
export VIP=<VIP address>
sudo ctr image pull ghcr.io/kube-vip/kube-vip:$KVVERSION
sudo ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip manifest pod \
    --arp \
    --controlplane \
    --address $VIP \
    --interface $INTERFACE \
    --enableLoadBalancer \
    --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

4. Initiate Kubernetes 1.29.0 control plane with 'kubeadm init', observe timeout and failure to initiate a cluster
5. Check logs for kube-vip container with crictl and observe an error

# crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs 72f0058318b86
time="2023-12-14T09:30:38Z" level=info msg="Starting kube-vip.io [v0.6.4]"
time="2023-12-14T09:30:38Z" level=info msg="namespace [kube-system], Mode: [ARP], Features(s): Control Plane:[true], Services:[false]"
time="2023-12-14T09:30:38Z" level=info msg="prometheus HTTP server started"
time="2023-12-14T09:30:38Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2023-12-14T09:30:38Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [lima-cp-1]"
I1214 09:30:38.362358       1 leaderelection.go:250] attempting to acquire leader lease kube-system/plndr-cp-lock...
E1214 09:30:38.907904       1 leaderelection.go:332] error retrieving resource lock kube-system/plndr-cp-lock: leases.coordination.k8s.io "plndr-cp-lock" is forbidden: User "kubernetes-admin" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"
E1214 09:30:40.258727       1 leaderelection.go:332] error retrieving resource lock kube-system/plndr-cp-lock: leases.coordination.k8s.io "plndr-cp-lock" is forbidden: User "kubernetes-admin" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-system"

Expected behavior
kube-vip is able to retrieve resource lock. This can be achieved by manually editing kube-vip.conf with following configuration before running kubeadm init.

  volumes:
  - hostPath:
      path: /etc/kubernetes/super-admin.conf
    name: kubeconfig

For details why see Kubernetes 1.29 CHANGELOG

Screenshots
n/a

Environment (please complete the following information):

• OS/Distro: Ubuntu 22.04 LTS
• Kubernetes Version: 1.29.0
• Kube-vip Version: 0.6.4

Kube-vip.yaml:
Will be provided if really needed.

Additional context
kubeadm init generates another kubeconfig file super-admin.conf in Kubernetes 1.29 and onwards which is required by kube-vip static pod to able start static pod during cluster initiation.

[tuxtof]

problem is that super-admin.conf is only present on the first control plane

looking deeper , it seems admin.conf have no permission at all, pretty strange

[louhisuo]

Not sure if it helps you forward but there is kubeadm issue #2414 which seems to describe this feature.

[neolit123]

this sounds like a bootstrap problem?

1.29 kubeadm init is doing the following:

• writes an admin.conf that has no binding to cluster-admin yet
• writes a super-admin.conf that has system:masters (super user)
• create cluster-admin binding using the super-admin

super-admin is not meant to be used by tools like kube-vip (..but see below)
if kube-vip needs permissiosn during setup it can, either:

• wait for admin.conf to receive permissions (not possible AFAIK, given the bootstrap sequence)
• use super-admin.conf during bootstrap and then moving to admin.conf

i recall we had a discussion that kube-vip must stop using the kubeadm admin* files, and should sign it's own scoped credentials. that is the best option, IMO.

[neolit123]

let's see what is actionable and we may have to also update these kubeadm docs:
https://github.com/kubernetes/kubeadm/blob/main/docs/ha-considerations.md#kube-vip

[chrischdi]

Workaround:

• use super-admin.conf during kubeadm init.
• revert the static pod back to admin.conf after successful kubeadm init (which causes a recreation of the kube-vip pod)

Example solution (only run on the node which runs kubeadm init):

• command pre-kubeadm:

  sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' \
            /etc/kubernetes/manifests/kube-vip.yaml
  

• command post-kubeadm (Edit note: this causes a pod restart and may cause flaky behavior):

  sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' \
            /etc/kubernetes/manifests/kube-vip.yaml
  

[prezha]

Workaround:

• use super-admin.conf during kubeadm init.
• revert the static pod back to admin.conf after successful kubeadm init (which causes a recreation of the kube-vip pod)

Example solution (only run on the node which runs kubeadm init):

• command pre-kubeadm:

  sed -i 's#path: /etc/kubernetes/admin.conf#path: /etc/kubernetes/super-admin.conf#' \
            /etc/kubernetes/manifests/kube-vip.yaml
  

• command post-kubeadm:

  sed -i 's#path: /etc/kubernetes/super-admin.conf#path: /etc/kubernetes/admin.conf#' \
            /etc/kubernetes/manifests/kube-vip.yaml
  

@chrischdi the above override should only apply to the hostPath and not the mountPath (that should stay /etc/kubernetes/admin.conf in either case)?

kube-vip/cmd/kube-vip.go

Line 153 in 3c0e5e3

kubeVipCmd.PersistentFlags().StringVar(&initConfig.K8sConfigFile, "k8sConfigPath", "/etc/kubernetes/admin.conf", "Path to the configuration file used with the Kubernetes client")