Talos 1.12.x and 1.13.x potentially break Hetzner hybrid clusters (VLANConfig)

[ArjonBu]

Hi guys, seems like something broke my setup when creating a fresh cluster on v1.12.7 on Hetzner hybrid (vSwitch).  The following updated config doesn't work either on 1.12.x nor 1.13.x.

Has anyone had this working? I tried everything but can't make it work. The worker nodes/masters on Cloud cannot reach out to the dedicated machines on vSwitch.  The vSwitch itself is fine because I tested it with Ubuntu. Does anyone have any idea what's up?

So, the dedicated machine joins K8s as a worker node, and it's ready, but it times out when trying to exec or see logs of a pod running on it. 100% connection issue. No firewalls on anything.

Full machineconfig.

machine:
  type: worker
  install:
    diskSelector:
      wwid: ${os_disk}
    image: ${installer}
    wipe: true
  kubelet:
    extraConfig:
      registerWithTaints:
        - key: node-role.kubernetes.io/storage
          effect: NoSchedule
    extraArgs:
      cloud-provider: external
      provider-id: hcloud://bm-${robot_server_id}
      rotate-server-certificates: true
    nodeIP:
      validSubnets:
        - ${vswitch_subnet_ipv4_cidr}
  kernel:
    modules:
      - name: nbd
  sysctls:
    net.core.somaxconn: "65535"
    net.core.netdev_max_backlog: "4096"
  features:
    diskQuotaSupport: true
    kubePrism:
      enabled: true
      port: 7445
    hostDNS:
      enabled: true
      forwardKubeDNSToHost: true
      resolveMemberNames: true
  systemDiskEncryption:
    ephemeral:
      provider: luks2
      keys:
        - nodeID: {}
          slot: 0
    state:
      provider: luks2
      keys:
        - nodeID: {}
          slot: 0
cluster:
  network:
    dnsDomain: cluster.local
    podSubnets:
      - ${cluster_pod_ipv4_cidr}
    serviceSubnets:
      - ${cluster_service_ipv4_cidr}
    cni:
      name: none
  proxy:
    disabled: true
---
apiVersion: v1alpha1
kind: HostnameConfig
hostname: ${hostname}
auto: off
---
apiVersion: v1alpha1
kind: LinkConfig
name: enp1s0
mtu: 1500
addresses:
  - address: $PUBLIC_IP/32
routes:
  - destination: $PUBLIC_GATEWAY/32
  - gateway: $PUBLIC_GATEWAY
---
apiVersion: v1alpha1
kind: VLANConfig
name: enp1s0.4001
vlanID: 4001
parent: enp1s0
mtu: 1400
addresses:
  - address: 10.0.2.3/24 # Private IP address
routes:
  - destination: 10.0.0.0/16 # Network CIDR
    gateway: 10.0.2.1 # vSwitch gateway
---
apiVersion: v1alpha1
kind: ResolverConfig
nameservers:
  - address: 1.1.1.1
  - address: 8.8.8.8
---
apiVersion: v1alpha1
kind: TimeSyncConfig
ntp:
  servers:
    - time.cloudflare.com
    - ntp1.hetzner.de
    - ntp2.hetzner.com
    - ntp3.hetzner.net

The old way of managing network has the same issue on 1.12.x

network:
    nameservers:
      - 1.1.1.1
      - 8.8.8.8
    interfaces:
      - interface: enp1s0
        dhcp: false
        mtu: 1500
        addresses:
          - ${public_ip_details.ip.ip}/32
        routes:
          - network: ${public_ip_details.ip.gateway}/32
          - network: 0.0.0.0/0
            gateway: ${public_ip_details.ip.gateway}
        vlans:
        - vlanId: ${vlan_id}
          dhcp: false
          mtu: 1400
          addresses:
            - ${private_ip}/24
          routes:
            - network: ${network_ipv4_cidr}
              gateway: ${vswitch_gateway}

✖ talosctl -n $PUBLIC_IP get links
NODE         NAMESPACE   TYPE         ID                VERSION   ALIAS   TYPE       KIND     HW ADDR                                           OPER STATE   LINK STATE
$PUBLIC_IP   network     LinkStatus   bond0             1                 ether      bond     ce:76:bd:fa:5e:a3                                 down         false
$PUBLIC_IP   network     LinkStatus   cilium_host       2                 ether      veth     0a:c2:9f:1b:9f:f9                                 up           true
$PUBLIC_IP   network     LinkStatus   cilium_net        2                 ether      veth     72:f3:1b:e1:65:cc                                 up           true
$PUBLIC_IP   network     LinkStatus   cilium_vxlan      1                 ether      vxlan    ca:e3:ed:51:a9:3e                                 unknown      true
$PUBLIC_IP   network     LinkStatus   dummy0            1                 ether      dummy    0a:7a:bc:0b:37:50                                 down         false
$PUBLIC_IP   network     LinkStatus   enp129s0          2                 ether               9c:6b:00:cb:2d:d0                                 down         false
$PUBLIC_IP   network     LinkStatus   enp1s0            3                 ether               98:b7:85:8a:66:24                                 up           true
$PUBLIC_IP   network     LinkStatus   enp1s0.4001       1                 ether      vlan     98:b7:85:8a:66:24                                 up           true
$PUBLIC_IP   network     LinkStatus   ip6tnl0           1                 tunnel6    ip6tnl   00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00   down         false
$PUBLIC_IP   network     LinkStatus   lo                1                 loopback            00:00:00:00:00:00                                 unknown      true
$PUBLIC_IP   network     LinkStatus   lxc59c724a9cefe   2                 ether      veth     26:5c:99:cf:9b:ed                                 up           true
$PUBLIC_IP   network     LinkStatus   lxc_health        2                 ether      veth     ce:e2:a3:8f:a3:7c                                 up           true
$PUBLIC_IP   network     LinkStatus   sit0              1                 sit        sit      00:00:00:00                                       down         false
$PUBLIC_IP   network     LinkStatus   teql0             1                 void                                                                  down         false
$PUBLIC_IP   network     LinkStatus   tunl0             1                 ipip       ipip     00:00:00:00                                       down         false

➜ talosctl -n $PUBLIC_IP get routes
NODE         NAMESPACE   TYPE          ID                                                             VERSION   DESTINATION                     GATEWAY          LINK              METRIC
$PUBLIC_IP   network     RouteStatus   252/inet4///0                                                  1                                                          lo                0
$PUBLIC_IP   network     RouteStatus   cilium_host/inet6//fe80::/64/256                               1         fe80::/64                                        cilium_host       256
$PUBLIC_IP   network     RouteStatus   cilium_net/inet6//fe80::/64/256                                1         fe80::/64                                        cilium_net        256
$PUBLIC_IP   network     RouteStatus   cilium_vxlan/inet6//fe80::/64/256                              1         fe80::/64                                        cilium_vxlan      256
$PUBLIC_IP   network     RouteStatus   enp1s0.4001/inet6//fe80::/64/256                               1         fe80::/64                                        enp1s0.4001       256
$PUBLIC_IP   network     RouteStatus   enp1s0/inet6//fe80::/64/256                                    1         fe80::/64                                        enp1s0            256
$PUBLIC_IP   network     RouteStatus   inet4//10.0.2.0/24/0                                           1         10.0.2.0/24                                      enp1s0.4001       0
$PUBLIC_IP   network     RouteStatus   inet4//172.20.11.151/32/0                                      1         172.20.11.151/32                                 cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4//$PUBLIC_IP_GW/32/1024                                   1         $PUBLIC_IP_GW/32                                 enp1s0            1024
$PUBLIC_IP   network     RouteStatus   inet4/10.0.2.1/10.0.0.0/16/1024                                1         10.0.0.0/16                     10.0.2.1         enp1s0.4001       1024
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.0.0/24/0                            1         172.20.0.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.1.0/24/0                            1         172.20.1.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.10.0/24/0                           1         172.20.10.0/24                  172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.11.0/24/0                           1         172.20.11.0/24                  172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.12.0/24/0                           1         172.20.12.0/24                  172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.14.0/24/0                           1         172.20.14.0/24                  172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.2.0/24/0                            1         172.20.2.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.3.0/24/0                            1         172.20.3.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.4.0/24/0                            1         172.20.4.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.5.0/24/0                            1         172.20.5.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.6.0/24/0                            1         172.20.6.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.7.0/24/0                            1         172.20.7.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.8.0/24/0                            1         172.20.8.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4/172.20.11.151/172.20.9.0/24/0                            1         172.20.9.0/24                   172.20.11.151    cilium_host       0
$PUBLIC_IP   network     RouteStatus   inet4//$PUBLIC_IP_GW//1024                                     2                                         $PUBLIC_IP_GW    enp1s0            1024
$PUBLIC_IP   network     RouteStatus   local/cilium_host/inet6//fe80::/128/0                          1         fe80::/128                                       cilium_host       0
$PUBLIC_IP   network     RouteStatus   local/cilium_host/inet6//fe80::8c2:9fff:fe1b:9ff9/128/0        1         fe80::8c2:9fff:fe1b:9ff9/128                     cilium_host       0
$PUBLIC_IP   network     RouteStatus   local/cilium_host/inet6//ff00::/8/256                          1         ff00::/8                                         cilium_host       256
$PUBLIC_IP   network     RouteStatus   local/cilium_net/inet6//fe80::/128/0                           1         fe80::/128                                       cilium_net        0
$PUBLIC_IP   network     RouteStatus   local/cilium_net/inet6//fe80::70f3:1bff:fee1:65cc/128/0        1         fe80::70f3:1bff:fee1:65cc/128                    cilium_net        0
$PUBLIC_IP   network     RouteStatus   local/cilium_net/inet6//ff00::/8/256                           1         ff00::/8                                         cilium_net        256
$PUBLIC_IP   network     RouteStatus   local/cilium_vxlan/inet6//fe80::/128/0                         1         fe80::/128                                       cilium_vxlan      0
$PUBLIC_IP   network     RouteStatus   local/cilium_vxlan/inet6//fe80::c8e3:edff:fe51:a93e/128/0      1         fe80::c8e3:edff:fe51:a93e/128                    cilium_vxlan      0
$PUBLIC_IP   network     RouteStatus   local/cilium_vxlan/inet6//ff00::/8/256                         1         ff00::/8                                         cilium_vxlan      256
$PUBLIC_IP   network     RouteStatus   local/enp1s0.4001/inet6//fe80::/128/0                          1         fe80::/128                                       enp1s0.4001       0
$PUBLIC_IP   network     RouteStatus   local/enp1s0.4001/inet6//fe80::9ab7:85ff:fe8a:6624/128/0       1         fe80::9ab7:85ff:fe8a:6624/128                    enp1s0.4001       0
$PUBLIC_IP   network     RouteStatus   local/enp1s0.4001/inet6//ff00::/8/256                          1         ff00::/8                                         enp1s0.4001       256
$PUBLIC_IP   network     RouteStatus   local/enp1s0/inet6//fe80::/128/0                               1         fe80::/128                                       enp1s0            0
$PUBLIC_IP   network     RouteStatus   local/enp1s0/inet6//fe80::9ab7:85ff:fe8a:6624/128/0            1         fe80::9ab7:85ff:fe8a:6624/128                    enp1s0            0
$PUBLIC_IP   network     RouteStatus   local/enp1s0/inet6//ff00::/8/256                               1         ff00::/8                                         enp1s0            256
$PUBLIC_IP   network     RouteStatus   local/inet4//10.0.2.255/32/0                                   1         10.0.2.255/32                                    enp1s0.4001       0
$PUBLIC_IP   network     RouteStatus   local/inet4//10.0.2.3/32/0                                     1         10.0.2.3/32                                      enp1s0.4001       0
$PUBLIC_IP   network     RouteStatus   local/inet4//127.0.0.0/8/0                                     1         127.0.0.0/8                                      lo                0
$PUBLIC_IP   network     RouteStatus   local/inet4//127.0.0.1/32/0                                    1         127.0.0.1/32                                     lo                0
$PUBLIC_IP   network     RouteStatus   local/inet4//127.255.255.255/32/0                              1         127.255.255.255/32                               lo                0
$PUBLIC_IP   network     RouteStatus   local/inet4//169.254.116.108/32/0                              1         169.254.116.108/32                               lo                0
$PUBLIC_IP   network     RouteStatus   local/inet4//172.20.11.151/32/0                                1         172.20.11.151/32                                 cilium_host       0
$PUBLIC_IP   network     RouteStatus   local/inet4//$PUBLIC_IP/32/0                                   1         $PUBLIC_IP/32                                     enp1s0            0
$PUBLIC_IP   network     RouteStatus   local/lo/inet6//::1/128/0                                      1         ::1/128                                          lo                0
$PUBLIC_IP   network     RouteStatus   local/lxc59c724a9cefe/inet6//fe80::/128/0                      1         fe80::/128                                       lxc59c724a9cefe   0
$PUBLIC_IP   network     RouteStatus   local/lxc59c724a9cefe/inet6//fe80::245c:99ff:fecf:9bed/128/0   1         fe80::245c:99ff:fecf:9bed/128                    lxc59c724a9cefe   0
$PUBLIC_IP   network     RouteStatus   local/lxc59c724a9cefe/inet6//ff00::/8/256                      1         ff00::/8                                         lxc59c724a9cefe   256
$PUBLIC_IP   network     RouteStatus   local/lxc_health/inet6//fe80::/128/0                           1         fe80::/128                                       lxc_health        0
$PUBLIC_IP   network     RouteStatus   local/lxc_health/inet6//fe80::cce2:a3ff:fe8f:a37c/128/0        1         fe80::cce2:a3ff:fe8f:a37c/128                    lxc_health        0
$PUBLIC_IP   network     RouteStatus   local/lxc_health/inet6//ff00::/8/256                           1         ff00::/8                                         lxc_health        256
$PUBLIC_IP   network     RouteStatus   lxc59c724a9cefe/inet6//fe80::/64/256                           1         fe80::/64                                        lxc59c724a9cefe   256
$PUBLIC_IP   network     RouteStatus   lxc_health/inet6//fe80::/64/256                                1         fe80::/64                                        lxc_health        256

talosctl pcap -i enp1s0.4001 during a ping from a cloud node shows only ARP requests going out from 10.0.2.3, reaching nothing and no ICMP traffic.

The weirdest thing is that when applying the machineconfig to the dedicated node, talosctl edit mc -n 10.0.2.3 actually works just one time. Then afterwards it stops.

Another detail is that when I run this command 3 times in a row, for 2 times I get the timeout instantly and the third time it takes a while to timeout.

➜ talosctl edit machineconfig -n 10.0.2.3 #instant
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.0.2.3:50000: i/o timeout"

✖  talosctl edit machineconfig -n 10.0.2.3 #instant
rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.0.2.3:50000: i/o timeout"

✖  talosctl edit machineconfig -n 10.0.2.3 #takes a while to timeout
^CSignal received, aborting, press Ctrl+C once again to abort immediately...
rpc error: code = Canceled desc = context canceled

talosctl cli works fine on all cloud nodes.

[smira]

Do you still have connectivity to the machine over a public IP? Is the problem only with the VLAN?

[smira]

Ok, if you have connectivity still, we can diagnose this further.

Talos 1.11->1.12 major change is the Linux bump 6.12->6.18, so this might be a driver issue.

You can try the following:

• talosctl pcap -i enp1s0 (the parent, with the VLAN-header pcap option / a privileged pod running tcpdump -e -i enp1s0 vlan 4001) — to see whether tagged frames are arriving on the wire at all.
• talosctl get ethernetstatus -n $PUBLIC_IP enp1s0 -o yaml — specifically the rx-vlan-filter / rx-vlan-hw-parse / tx-vlan-hw-insert feature lines and the NIC driver name.

You can try disabling the VLAN offloading features on the NIC to see if it fixes your issue.

[ArjonBu]

I recall that I upgraded from 1.11 to 1.12 and it worked. When I reinstalled the cluster from scratch it didn't.

talosctl pcap -i enp1s0: Not sure what to look for here. I do see a lot of traffic but can't tell.

✖  talosctl -n $PUBLIC_IP get ethernetstatus enp1s0 -o yaml
node: $PUBLIC_IP
metadata:
    namespace: network
    type: EthernetStatuses.net.talos.dev
    id: enp1s0
    version: 2
    owner: network.EthernetStatusController
    phase: running
    created: 2026-05-15T10:31:03Z
    updated: 2026-05-15T10:32:49Z
spec:
    linkState: true
    port: DirectAttach
    duplex: Full
    ourModes:
        - FIBRE
        - 10000baseT/Full
        - Pause
    rings:
        rx-max: 8192
        tx-max: 8192
        rx: 512
        tx: 512
        tx-push: false
        rx-push: false
    features:
        tx-scatter-gather: on
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: on
        tx-checksum-ipv6: off [fixed]
        highdma: on [fixed]
        tx-scatter-gather-fraglist: off [fixed]
        tx-vlan-hw-insert: on
        rx-vlan-hw-parse: on
        rx-vlan-filter: on
        vlan-challenged: off [fixed]
        tx-generic-segmentation: on
        rx-gro: on
        rx-lro: off
        tx-tcp-segmentation: on
        tx-gso-robust: off [fixed]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: on
        tx-fcoe-segmentation: off [fixed]
        tx-gre-segmentation: on
        tx-gre-csum-segmentation: on
        tx-ipxip4-segmentation: on
        tx-ipxip6-segmentation: on
        tx-udp_tnl-segmentation: on
        tx-udp_tnl-csum-segmentation: on
        tx-gso-partial: on
        tx-tunnel-remcsum-segmentation: off [fixed]
        tx-sctp-segmentation: off [fixed]
        tx-esp-segmentation: on
        tx-udp-segmentation: on
        tx-gso-list: off [fixed]
        tx-tcp-accecn-segmentation: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: on
        rx-ntuple-filter: off
        rx-hashing: on
        rx-checksum: on
        tx-nocache-copy: off
        loopback: off [fixed]
        rx-fcs: off [fixed]
        rx-all: off
        tx-vlan-stag-hw-insert: off [fixed]
        rx-vlan-stag-hw-parse: off [fixed]
        rx-vlan-stag-filter: off [fixed]
        l2-fwd-offload: off
        hw-tc-offload: off
        esp-hw-offload: on
        esp-tx-csum-hw-offload: on
        rx-udp_tunnel-port-offload: off [fixed]
        tls-hw-tx-offload: off [fixed]
        tls-hw-rx-offload: off [fixed]
        rx-gro-hw: off [fixed]
        tls-hw-record: off [fixed]
        rx-gro-list: off
        macsec-hw-offload: off [fixed]
        rx-udp-gro-forwarding: off
        hsr-tag-ins-offload: off [fixed]
        hsr-tag-rm-offload: off [fixed]
        hsr-fwd-offload: off [fixed]
        hsr-dup-offload: off [fixed]
    channels:
        other-max: 1
        combined-max: 20
        other: 1
        combined: 20

I disabled it but it doesn't fix the issue

  tx-vlan-hw-insert: false
  rx-vlan-hw-parse: false
  rx-vlan-filter: false

[ArjonBu]

Passing this to an AI, it says shows only Ethernet → IPv4 → TCP/ARP, with no separate VLAN/Dot1Q layer anywhere.

[ArjonBu]

I feel stupid but cilium was using the wrong interface (vswitch ID changed). Nothing related to Talos