Upgrade hcloud-cloud-controller-manager following Hetzner's notice.

[wassup-]

The mail received from Hetzner:

We would like to inform you that you must update your hcloud-cloud-controller-manager by 30 June 2026, at the latest.

What should you do?

Please update your hcloud-cloud-controller-manager to the latest available version, or at least to v1.30.1, to ensure uninterrupted operation starting 1 July 2026.

Why is an update required?

We are discontinuing support for the "datacenter" API property in favor of "location" for Primary IPs and Servers (see changelog: https://docs.hetzner.cloud/changelog#2025-12-16-phasing-out-datacenters).

This change affects all existing installations of the hcloud-cloud-controller-manager that are using a version older than v1.30.1. Once the API change takes effect, the controller will continue to attempt to access the deprecated "datacenter" property. This will cause the controller to crash within a few minutes.

In this state, the hcloud-cloud-controller-manager can no longer correctly initialize new Kubernetes nodes, remove nodes whose underlying server no longer exists from the cluster, or apply changes to Load Balancer configurations via Kubernetes service objects.

So my question is, what is the recommended / correct way to upgrade the hcloud-cloud-controller-manager when using this project?

For what it's worth: I am currently on version 2.17.1 of this project, running terraform plan after upgrading to 2.19.3 results in the following error:

│ Error: Provider produced invalid plan
│ 
│ Provider "provider[\"registry.opentofu.org/hetznercloud/hcloud\"]" planned
│ an invalid value for
│ module.kube-hetzner.hcloud_load_balancer_network.cluster[0].ip: planned
│ value cty.UnknownVal(cty.String) does not match config value
│ cty.StringVal("10.255.0.1") nor prior value cty.StringVal("10.255.0.1").

[abriginets]

I just went through this upgrade (2.16.0 -> 2.19.3, ARM control plane cax11 nodes, Flannel CNI, Traefik, cluster autoscaler). Here's everything that broke and how I fixed it.

Provider produced invalid plan
planned value cty.UnknownVal(cty.String) does not match config value
cty.StringVal("10.255.0.1") nor prior value cty.StringVal("10.255.0.1")

It's a state conflict due to subnet change between module versions. The module changed the LB network subnet. The old LB attachment has IP 10.255.0.1 (on the 10.255.0.0/16 subnet), but the new module wants to place the LB on the agent subnet 10.0.0.0/16. The provider cannot reconcile this.

What I did:

Caution

This causes downtime, plan accordingly.

1. Detached the LB from the private network manually via Hetzner Cloud Console (Load Balancer -> Networking tab -> Private Networks -> Click on 3 dots -> Detach)
2. Removed the stale resource from Terraform state: terraform state rm 'module.kube-hetzner.hcloud_load_balancer_network.cluster[0]'
3. Ran terraform apply and it recreated the attachment cleanly on the correct subnet

Your LB might not work properly after this because of the missing targets. Run this on your cluster:

kubectl rollout restart deployment/hcloud-cloud-controller-manager -n kube-system

The new CCM pod will pick up the Traefik service annotations and sync the LB targets. For me traffic was restored immediately (got a notification from uptimerobot).

terraform apply renamed eth1 to enp7s0

This is the most dangerous part of the upgrade. When Terraform reattaches the private network (or re-provisions anything that causes a network change), MicroOS applies predictable interface naming, renaming eth1 to enp7s0. k3s uses eth1 as the flannel interface and the node IP source, so after the rename kubectl stopped working (along with Lens), etcd lost quorum and all 3 control planes went into NotReady.

What I did:

SSH-ed into each control plane node:

# Find the MAC address of the private network interface
ip a  # note the MAC of enp7s0 (or whatever it was renamed to)

# Create a persistent udev rule to force the name back to eth1
cat > /etc/udev/rules.d/70-persistent-net.rules <<EOF
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="MAC_ADDRESS_HERE", NAME="eth1"
EOF

# Reboot the node
reboot

After reboot, eth1 comes back, k3s starts, and the node rejoins. Do this sequentially, wait for each node to come back Ready before doing the next, to maintain etcd quorum.

Hope this helps!

[ajayyy]

Upgraded from from 2.11.7 and the only change needed was this, thanks!

Here's how you can automate this in terraform

resource "null_resource" "network-update" {
  for_each = local.agent_nodes

  connection {
    user           = "root"
    private_key    = var.ssh_private_key
    agent_identity = local.ssh_agent_identity
    host           = module.agents[each.key].ipv4_address
    port           = var.ssh_port
  }

  provisioner "remote-exec" {
    inline = [
      "cat > /etc/udev/rules.d/70-persistent-net.rules <<EOF\nSUBSYSTEM==\"net\", ACTION==\"add\", ATTR{address}==\"$(ip addr show enp7s0 | awk '/link\\/ether/ {print $2}')\", NAME=\"eth1\"\nEOF",
      "reboot",
    ]
  }

  triggers = {
    always_run = "${timestamp()}"
  }
}