Talos on OVHCloud dedicated servers

[Biscottex]

Hello,

I am trying to deploy Talos on a OVHCloud dedicated server using a vRack (OVH L2 private Network).
I have multiple error and I am a bit lost, I noticed some of you succeed of deploying a Talos cluster on OVH and might be able to help.

I tried with Talos 0.12.5, 0.12.6, 0.13.0, it is already a while that I am trying to deploy Talos on OVH.

Installation

I have the following server (SYS-3 | Intel Xeon-E 2288G) and picked the following Talos configuration:

customization:
    extraKernelArgs:
        - ip=dhcp,mac,xx:xx:xx:40:52:52
    systemExtensions:
        officialExtensions:
            - siderolabs/i915
            - siderolabs/intel-ice-firmware
            - siderolabs/intel-ucode
            - siderolabs/realtek-firmware

After filling https://factory.talos.dev, I have a qcow2 link.

I install the server with the OVH UI (Bring Your Own Image) where I provided the qcow2 link.
In the section "Path of the EFI bootloader from the OS installed on the server", I tried:

• /EFI/BOOT/BOOTX64.EFI
  and
• \\EFI\\BOOT\\BOOTX64.EFI

Issue 1: Failed to boot (EFI).

At boot, the server is stuck in a reboot loop.
When I use the remote KVM, I can see the logs and it shows:

Trying to boot from  /EFI/BOOT/BOOTX64.EFI (or \\EFI\\BOOT\\BOOTX64.EFI)
Boot from SAN device 0x00 failed: Error 0x3d222083 (https://ipxe.org/3d222083)
Failed to boot with particular efi file name, trying to guess.

Afterwards it displays "failed to determine platform".

Networking info

To go further, in the boot menu (KVM), I choose: "Boot Fallback boot loader from EFI".
Talos is now booting in "STAGE: Maintenance"

vRack & networking info

The vRack is a private network supporting L2. There is not DHCP support as far as I know.

Server info

ipv4: xxx.xx.30.99
ipv4 Gateway: xxx.xx.30.254
ipv6: xxxx:xxxx:303:fa63::/64
ipv6 Gateway: xxxx:xxxx:0303:faff:00ff:00ff:00ff:00ff

Network interface controller

public interface with DHCP: xx:xx:97:b0:56:e7
private interface (vRack): xx:xx:97:b0:56:e8

vRack IP address:

I have a free IPv6 address linked to the vRack:

• xxxx:xxxx:d00:b600::/56

Info from Debian

ip link show

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether xx:xx:97:b0:56:e7 brd ff:ff:ff:ff:ff:ff
    altname enxxxxx97b056e7
3: enxxxxx7a096bb8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether xx:xx:7a:09:6b:b8 brd ff:ff:ff:ff:ff:ff
4: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether xx:xx:97:b0:56:e8 brd ff:ff:ff:ff:ff:ff
    altname enxxxxx97b056e8

ip route show default
default via xxx.xx.30.254 dev enp3s0f0 proto dhcp src xxx.xx.30.99 metric 100

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether xx:xx:97:b0:56:e7 brd ff:ff:ff:ff:ff:ff
    altname enxxxxx97b056e7
    inet xxx.xx.30.99/24 metric 100 brd 141.94.30.255 scope global dynamic enp3s0f0
       valid_lft 85174sec preferred_lft 85174sec
    inet6 xxxx:xxxx:303:fa63::/64 scope global
       valid_lft forever preferred_lft forever
    inet6 xxxx::xxxx:97ff:feb0:56e7/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever
3: enxxxxx7a096bb8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether xx:xx:7a:09:6b:b8 brd ff:ff:ff:ff:ff:ff
4: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether xx:xx:97:b0:56:e8 brd ff:ff:ff:ff:ff:ff
    altname enxxxxx97b056e8

Issue 2: DHCP error (in maintenance mode)

DHCP request/renew failed... unable to received an offer...,"link": "enp3s0f1".
(enp3s0f1 is the network card linked to the vRack.)

I then tried to specify the network interface (public) as default DHCP interface as a Kernel argument during the factory.talos.dev configuration:
ip=dhcp,mac,xx:xx:xx:40:52:52
or
ip=:::::enxxxxx97b056e7:dhcp

I am still facing the same issue but it might be due to the Talos maintenance mode.

patch.yaml (still in maintenance mode)

I applied the patch.yaml in maintenance mode. In maintenance mode, I am able to ping the server.
Is it fine to apply a static IPv6 to the server like this? xxxx:xxxx:d00:b600::0003/56

cluster:
  allowSchedulingOnControlPlanes: true
  network:
    cni:
      name: none
  proxy:
    disabled: true

machine:
  network:
    nameservers:
      - 213.186.33.99
      - 213.251.188.146
      - 213.251.188.141
      - 8.8.8.8
      - 1.1.1.1

    interfaces:
      # Public interface (DHCP)
      - interface: enp3s0f0
        dhcp: true
      # Private vRack interface (STATIC IPv6)
      - interface: enp3s0f1
        dhcp: false
        addresses:
          - "xxxx:xxxx:d00:b600::0003/56"
        mtu: 1500

L2 IPPool question (a bit out of scope)

I am using cilium, which cidr should I use xxxx:xxxx:d00:b600::/56? I only have two servers in the vRack.
I am not there yet as I have installation problem for the moment (Issue 1 & 2).

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: vrack-pool
spec:
  #reserve the first and last IPs of CIDRs so we do not use them.
  allowFirstLastIPs: "No"
  blocks:
  - cidr: "xxxx:xxxx:d00:b600::/56"
  serviceSelector:
    matchLabels:
      lb-type: vrack-internal
---
apiVersion: "cilium.io/v2alpha1"
kind: CiliumL2AnnouncementPolicy
metadata:
  name: l2-announcement-policy
spec:
  serviceSelector:
    matchLabels:
      io.kubernetes.service.namespace: xxx-ns
  interfaces:
    - enp3s0f1
  externalIPs: true
  loadBalancerIPs: true
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux

[smira]

I can only comment on Talos-specific things here:

Issue 2: DHCP error (in maintenance mode)

You don't need to worry about this at all. Don't put any kernel args, Talos defaults to running DHCP, so it will run DHCP, get an IP so that you can connect to the machine. When connected, you can finish configuring your machine which will stop DHCP on the interfaces you don't want to run DHCP on.

Talos 1.12+ provides a new way to configure networking (https://docs.siderolabs.com/talos/v1.13/networking/configuration/overview), but probably in your case it's something like:

apiVersion: v1alpha1
kind: DHCPv4Config
name: enp3s0f0
---
apiVersion: v1alpha1
kind: LinkConfig
name: enp3s0f1
addresses:
  - address: "xxxx:xxxx:d00:b600::0003/56"  

Applying explicit config will stop auto-DHCP, and will only run explicit one on enp3s0f0.

[Biscottex]

Since your previous message, talos is now able to boot. It bootstraps but did not became ready.
I suspect a network misconfiguration on my side, I tried multiple configurations without success.

talosctl -n xxx.xx.30.99 logs controller-runtime gives this error:

xxx.xx.30.99: DEBUG restarting controller in 543.274588ms {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController"}
xxx.xx.30.99: DEBUG controller starting {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController"}
xxx.xx.30.99: DEBUG serving static pod manifests {"component": "controller-runtime", "controller": "k8s.StaticPodServerController", "size": 12444}
xxx.xx.30.99: ERROR controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: an error on the server ("Authorization error (user=apiserver-kubelet-client, verb=get, resource=nodes, subresource=pods)") has prevented the request from succeeding"}

talosctl health gives:

waiting for all k8s nodes to report: Get "https://xxx.xx.30.99:6443/api/v1/nodes": dial tcp 141.94.30.99:6443: connect: connection refused

In talosctl, I utilize an ipv4 (endpoint) and I want talos to use the private network (OVHCloud vrack) with an ipv6 subnet for his communication within the cluster.
In other word, the cluster (etcd,..) will use the private vrack.

apiVersion: v1alpha1
kind: DHCPv4Config
name: enp3s0f0
---
apiVersion: v1alpha1
kind: LinkConfig
name: enp3s0f1
# https://docs.siderolabs.com/talos/v1.13/reference/configuration/network/linkconfig
# Configure addresses to be statically assigned to the link.
addresses:
  - address: "xxxx:xxxx:d00:b600::3/64" # IP address to be assigned to the link in /64.

I added this config so etcd and the Kubelet will use the private network:

cluster:
  allowSchedulingOnControlPlanes: true
  etcd:
    advertisedSubnets: # listenSubnets defaults to advertisedSubnets if not set explicitly
      - xxxx:xxxx:d00:b600::/64

machine:
  features:
    kubePrism:
      enabled: true
      port: 7445
  network:
    nameservers:
      - 213.186.33.99
      - 213.251.188.146
      - 213.251.188.141
      - 8.8.8.8
      - 1.1.1.1

  #We want to use the private network for etcd and kubelet communication
  kubelet:
    nodeIP:
      validSubnets:
        - xxxx:xxxx:d00:b600::/64

I can ping the IPv6 Gateway of the private network

talosctl -n xxx.xx.30.99 debug nicolaka/netshoot
ping6 xxxx:xxxx:d00:b600::1

PING xxxx:xxxx:d00:b600::1 (xxxx:xxxx:d00:b600::1) 56 data bytes
64 bytes from xxxx:xxxx:d00:b600::1: icmp_seq=1 ttl=64 time=0.525 ms
64 bytes from xxxx:xxxx:d00:b600::1: icmp_seq=2 ttl=64 time=0.360 ms
64 bytes from xxxx:xxxx:d00:b600::1: icmp_seq=3 ttl=64 time=0.289 ms

I can ping myself too in the vrack.
ping6 xxxx:xxxx:d00:b600::3

PING xxxx:xxxx:d00:b600::3 (xxxx:xxxx:d00:b600::3) 56 data bytes
64 bytes from xxxx:xxxx:d00:b600::3: icmp_seq=1 ttl=64 time=0.016 ms
64 bytes from xxxx:xxxx:d00:b600::3: icmp_seq=2 ttl=64 time=0.065 ms
64 bytes from xxxx:xxxx:d00:b600::3: icmp_seq=3 ttl=64 time=0.057 ms

talosctl -n xxx.xx.30.99 get links

NODE           NAMESPACE   TYPE         ID         VERSION   ALIAS   TYPE       KIND     HW ADDR                                           OPER STATE   LINK STATE
xxx.xx.30.99   network     LinkStatus   bond0      1                 ether      bond     ce:61:68:83:92:91                                 down         false
xxx.xx.30.99   network     LinkStatus   dummy0     1                 ether      dummy    42:90:f5:9d:33:a5                                 down         false
xxx.xx.30.99   network     LinkStatus   enp3s0f0   6                 ether               fc:34:97:b0:56:e7                                 up           true
xxx.xx.30.99   network     LinkStatus   enp3s0f1   5                 ether               fc:34:97:b0:56:e8                                 up           true
xxx.xx.30.99   network     LinkStatus   ip6tnl0    1                 tunnel6    ip6tnl   00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00   down         false
xxx.xx.30.99   network     LinkStatus   lo         2                 loopback            00:00:00:00:00:00                                 unknown      true
...

talosctl -n xxx.xx.30.999 get addresses

NODE           NAMESPACE   TYPE            ID                                      VERSION   ADDRESS                        LINK
xxx.xx.30.99   network     AddressStatus   enp3s0f0/xxx.xx.30.99/24                1         xxx.xx.30.99/24                enp3s0f0
xxx.xx.30.99   network     AddressStatus   enp3s0f0/fe80::fe34:97ff:feb0:56e7/64   2         fe80::fe34:97ff:feb0:56e7/64   enp3s0f0
xxx.xx.30.99   network     AddressStatus   enp3s0f1/xxx:xxx:d00:b600::3/64        2         xxx:xxx:d00:b600::3/64        enp3s0f1
xxx.xx.30.99   network     AddressStatus   enp3s0f1/fe80::fe34:97ff:feb0:56e8/64   2         fe80::fe34:97ff:feb0:56e8/64   enp3s0f1
xxx.xx.30.99   network     AddressStatus   lo/127.0.0.1/8                          1         127.0.0.1/8                    lo
xxx.xx.30.99   network     AddressStatus   lo/169.254.116.108/32                   1         169.254.116.108/32             lo
xxx.xx.30.99   network     AddressStatus   lo/::1/128                              1         ::1/128                        lo

vrack ipv6 Gateway: xxxx:xxxx:d00:b600::1 (OVH vrack IPV6 doc)

vrack ipv6 assigned to Talos by the config: xxxx:xxxx:d00:b600::3
Should I specify the Gateway in the LinkConfig?
Or anyone has an example of a private ipv6 network configuration for OVH?