Support for mixing regions and private networks (to circumvent 100 servers limit)

[knuurr]

Description

Limit of 100 servers in total in single hetzner private network makes it hard to run large scale clusters because there is a limit of 100 serves total in private network:

Hetzner limits to max 100 servers TOTAL in a private network. I am extremely disappointed because this makes it impossible to create large clusters with more than 100 nodes.

(see https://lowendtalk.com/discussion/187187/very-disappointing-limitation-in-hetzner-cloud-max-100-servers-per-private-network)

On top of that, if one wants to schedule nodes from outside network zone that's being used as "core" for cluster (i.e all nodes come from Germany+Finland region, but someone wants a US based node) provisoning will fail because this node won't be able to communicate using private IP (which usage of seem to be hardcoded in project)

I suggest there should be some mechanism where we can specify multiple private networks to use for scheduling nodes (i.e as option for particular nodes, along count, label and other).
These private networks could be either created on-demand with kube-hetzner as there is defined node pool which uses it, pre-created externally outside kube-hetzner, or even maybe as list of private networks to use and managed under kube-hetzner project

For nodes from outside networking zone where core infra resides, some kind of overlay network would have to be enforced, and public control plane LB IP address used for node joining instead of private IP. Use of network overlay could be used as boolean somewhere in kube.tf and if set to true, settings in CNI could be used accordingly. To denote that given node pool is "external" (external to the core where cluster resides) it could, again, be denoted using some property like external= true along count, label etc.

The one thing I didn't think about is what about cluster autoscaler. Could cluster autoscaler schedule nodes on demand in more that one private network? It would have to be aware when limit is hit, and maybe there would have to be some preference list which private networks to prioritize.

That's just very high level idea that comes fro my experience with this project. Again, great project, however I met some edge scenarios which aren't handled.

Happy to discuss this further!

[asemarian]

You seem to have an interesting use case. I don't think that's a limitation with this project or with the cloud provider. Kubernetes clusters that span multiple regions is probably not a good idea mainly due to network latency.

[knuurr]

You seem to have an interesting use case. I don't think that's a limitation with this project or with the cloud provider. Kubernetes clusters that span multiple regions is probably not a good idea mainly due to network latency.

In OP I linked a complain where user says no more than 100 servers can be at once in hetzner's virtual network

Limits

Up to 1 vSwitch per Network
Up to 50 subnets per Network
Up to 100 routes per Network
Up to 100 attached resources (servers & Load Balancers) per Network
Up to 3 networks per server
Up to 5 alias IPs in addition to the private main IP per server

https://docs.hetzner.com/networking/networks/overview/

So yes there definietly are some limitations on hetzner side.
My idea is to overcome this - say, use multiple virtual networks. Tbh when I wrote it I assumed they can have private routing so latency wouldn't be so bad in this scenario. I don't think this project currently supports more than single virtual network, or any other solution which would help overcome this limitation.

Kubernetes clusters that span multiple regions is probably not a good idea mainly due to network latency.

That's true, I agree. But I nod need may of such nodes. I am ok with that in my scenario because of very specific needs.

I'd just like to have at least some choice for all of these.

[mysticaltech]

Yes folks, this is an important limitations, we need to support multiple networks. Will address soon for the upcoming v3.

[knuurr]

Thank you @mysticaltech . I wonder is there any ETA?

[knuurr]

I've recently seen that another project, hetzner-k3s supports clusters above 100 nodes - and the tech stack is similar (there is CA too)

Large Production Clusters (50-200+ nodes)
Use the large cluster configuration shown above with: - Multiple node pools for different workloads - Custom firewall and IP query server - Larger instance types for masters - External datastore if needed

I wonder if you bave both idea how to do it and will, or just will, because answer might lie in how this project is doing it. sadly I didn't have time (yet) to reverse engineer how it's done there but I maybe hope to find some work hours.

[mysticaltech]

This is a great feature idea for large-scale deployments! Multi-region support with multiple private networks would enable clusters beyond the 100-server limit.

This is a significant architectural change that needs:

• RFC-style design discussion
• Autoscaler integration planning
• Overlay network considerations

Converting to a Discussion for proper community input.

→ Please create a discussion at: https://github.com/mysticaltech/terraform-hcloud-kube-hetzner/discussions/new?category=ideas

This will allow the community to contribute ideas and we can design this together for a future release.