Private network for bare metal host / cloud hybrid

[pucilpet]

/kind feature

Describe the solution you'd like
It is possible to link bare metal servers to the same private network as the cloud servers. I think it would be good to have an option for the private networking for hybrid clusters.

Anything else you would like to add:
More info: https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch/

Adding the vswitch is possible with the robot web service:
https://robot.your-server.de/doc/webservice/en.html#post-vswitch

[batistein]

This feature is not implemented yet. Since we had several problems with the private network in the past (very high ping, interruptions, etc.), we do not use it for our offers, so it has less prio from our side.
But I would love to see any contribution on it.

[shurshun]

Hi there! I would like to say thank you for such a handy product! I would also like to agree with the topic starter on the issue of private network support: we plan to order a set-up with a private network between servers (without the cloud) and, unfortunately, this is the only factor limiting the use of this provider :(

[mads-sogaard]

Hi just for my understanding, is it possible to use CAPI to provision and manage a bare metal only cluster, and then use a dedicated switch as private network (which can also be ordered from Hetzner)? I know pucilpet asks for a hybrid setup :-)

[batistein]

@mads-sogaard we didn't implemented the private network stuff on Hetzner Dedicated. In our production cluster we use the konnectivity-service, Cilium Host Firewall and run mTLS everywhere so we do not have the use case for private network. But of course anyone is welcome to contribute a PR.

[mstarostik]

I also figured the private networks aren't of that much use unless and until they offer more advanced routing setup between different private networks, NAT to public or the likes.
Anyway, one thing using a private network turned out to be still very useful to me is so I can get a known (private) IP for the API Server LB to pass as k8sServiceHost to Cilium when enabling its kube-proxy replacement. I'd be happy to simplify the setup some but haven't figured a way to pull the public IP for the LB into the ArgoCD app that provisions Cilium on newly created clusters.

[batistein]

@mstarostik I would recommend to use a domain in a production setup, this way you also solve your problem because you always know the k8sServiceHost in advance.

[mstarostik]

@batistein this might help a little, however it shifts the problem from
provision LB and nodes => get their IP(s) into config => provision Cilium
to
provision LB and nodes => get their IP(s) into DNS => provision Cilium
the middle part is not something easily done when going for a purely declarative setup. ExternalDNS might help when using a DNS name, but isn't up and running at the time the CNI config needs the info already. There sure are ways to implement this, but for my setup the stable and pre-known API server private IP helps a lot.
YMMV for sure