hostAliases do not work as static pod manifests in Kubernetes v1.29

[chrischdi]

Describe the bug

Created a cluster with CAPV and Kubernetes v1.29 and kube-vip as static pod

To Reproduce

• Create a cluster using Kubernetes v1.29 and kube-vip, via kubeadm
• See the Kube-VIP to not get healthy and not propagating the VIP
  erver running also as static pod (using kubeadm via Cluster API):

Expected behavior

Kube-VIP to successfully connect to the Kube-APIServer running on the same host and propagating the VIP.

Screenshots

Log lines showing kube-vip trying to resolve kubernetes via dns (replaced parts of the DNS IP to x.x.x:

E1220 08:24:50.720834       1 leaderelection.go:332] error retrieving resource lock kube-system/plndr-cp-lock: Get "https://kubernetes:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/plndr-cp-lock": dial tcp: lookup kubernetes on x.x.x.1:53: no such host

Static pod manifest (replaced address to x.x.x.x):

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  name: kube-vip
  namespace: kube-system
spec:
  containers:
  - args:
    - manager
    env:
    - name: vip_arp
      value: "true"
    - name: port
      value: "6443"
    - name: vip_cidr
      value: "32"
    - name: cp_enable
      value: "true"
    - name: cp_namespace
      value: kube-system
    - name: vip_ddns
      value: "false"
    - name: vip_leaderelection
      value: "true"
    - name: vip_leasename
      value: plndr-cp-lock
    - name: vip_leaseduration
      value: "15"
    - name: vip_renewdeadline
      value: "10"
    - name: vip_retryperiod
      value: "2"
    - name: address
      value: x.x.x.x
    - name: prometheus_server
      value: :2112
    image: ghcr.io/kube-vip/kube-vip:v0.6.4
    imagePullPolicy: Always
    name: kube-vip
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
    volumeMounts:
    - mountPath: /etc/kubernetes/admin.conf
      name: kubeconfig
  hostAliases:
  - hostnames:
    - kubernetes
    ip: 127.0.0.1
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/admin.conf
    name: kubeconfig
status: {}

Hosts file of the pod:

root@capi-quickstart-7hvtl:/home/capv# ps -ax | grep kube-vip
   3452 ?        Ssl    0:00 /kube-vip manager
   3516 pts/1    S+     0:00 grep --color=auto kube-vip
root@capi-quickstart-7hvtl:/home/capv# cat /proc/3452/root/etc/hosts
::1         ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6
127.0.0.1   capi-quickstart-7hvtl capi-quickstart-7hvtl localhost localhost.localdomain localhost4 localhost4.localdomain4

Environment (please complete the following information):

• OS/Distro: Ubuntu 22.04
• Kubernetes Version: v1.29.0
• Kube-vip Version: 0.6.4

Kube-vip.yaml:
If Possible add in your kube-vip manifest (please remove anything that is confidential)

Additional context

pod.spec.hostAliases are not expected to work on hostNetwork pods:

HostAliases is an optional list of hosts and IPs that will be injected into
the pod's hosts file if specified. This is only valid for non-hostNetwork
pods.
HostAlias holds the mapping between IP and hostnames that will be injected
as an entry in the pod's hosts file.

Source: kubectl explain pod.spec.hostAliases

[chrischdi]

When manually adding the line 127.0.0.1 kubernetes to the hosts file, the next issue we hit is #684

[neolit123]

pod.spec.hostAliases are not expected to work on hostNetwork pods:

this seems like a problem in the spec of kube-vip, maybe there is a workaround.

[chrischdi]

Possible workaround: use a hand-crafted hosts file and mount it via a hostPath volume instead of relying on hostAliases.

Regarding:

pod.spec.hostAliases are not expected to work on hostNetwork pods:

this seems like a problem in the spec of kube-vip, maybe there is a workaround.

I'm not sure, I did use the very same spec in v1.28 and it did work there. I'd guess they fixed something in kubelet, so hostAliases don't work anymore (as outlined in the description) for hostNetwork pods. However I was not able to directly find an issue or pull request related to that.

[neolit123]

I'm not sure, I did use the very same spec in v1.28 and it did work there. I'd guess they fixed something in kubelet, so hostAliases don't work anymore (as outlined in the description) for hostNetwork pods. However I was not able to directly find an issue or pull request related to that.

the information in the description could be completely out of date and inaccurate.
i will ping #sig-node and/or network to see if someone knows.
UPDATE: https://kubernetes.slack.com/archives/C09QYUH5W/p1703068817039879

@thebsdbox did the hostAliases actually work in the kube-vip manifest, it seems they were in there since epoch?

[neolit123]

i wasn't able to reproduce this locally with a bare kubeadm setup. @chrischdi confirmed similar with kind. so it seems like a problem in the test CAPI cluster.

[neolit123]

i wasn't able to reproduce this locally with a bare kubeadm setup. @chrischdi confirmed similar with kind. so it seems like a problem in the test CAPI cluster.

k/k ticket to understand if the docs are wrong:
kubernetes/kubernetes#122420

[chrischdi]

Workaround for now: instead of hostAliases:

• create a file (e.g. /etc/kubernetes/manifests/kube-vip.yaml) with the following content:

  127.0.0.1 localhost kubernetes
  

• Add the following to the static pod manifest:

  spec:
    containers:
    - ...
      volumeMounts:
      ...
      - mountPath: /etc/hosts
        name: etchosts
    ...
    volumes
      ...
      - hostPath:
          path: /etc/kube-vip.hosts
          type: File
        name: etchosts

Alternatively, other ips instead of the dns name kubernetes could get used, but that requres a different kubeconfig or fixes in kube-vip.

[tuxtof]

hostAliases was needed and working perfectly until 1.28

start to break with 1.29, and only when cloud-provider is set to external like with CAPV in the issue or with CAPX in my case

main issue seems here kubernetes/kubernetes#122420 (comment)