-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for bare-metal workers #113
Comments
The bare metal support would be highly appreciated. A label, that causes CCM to ignore bare metal nodes, would be fine as an intermediate step. This would make CCM still functional and useful in the meantime. |
Additional (already closed) issues: There are a few problems with adding dedicated servers are real "nodes" to the k8s cluster.
We will look into how we can improve it, but I can not promise something. |
Any update on this? |
@ctodea you can have a look to this https://github.com/cluster-api-provider-hcloud/hcloud-cloud-controller-manager |
@ctodea we managed to get a cluster working, where most nodes, including the master, are cloud servers. And some nodes are root servers, e.g. for databases. Basically the root servers should be mostly ignored by the CCM and CSI plugin. Maybe this helps:
You need to connect the root servers via vSwitch, though. Maybe #172 results in a mainline solution ... |
Many thanks for the update @malikkirchner @batistein |
Hi @malikkirchner But you can't create a route 10.240.1.0/24 via 10.240.1.2 in api. Then how will the communication between the pods of the 10.240.0.0/24 and 10.240.1.0/24 network work? |
Hi @identw, that is an excellent point, I do not know and was wondering myself. According to #133 (comment) that should never have worked. We are using kubeadm to setup the cluster and Cilium as CNI plugin. I am happy to share the exact config, if you are interested. I have two guesses how this can 'work'. Either the vSwitch does some routing, that I do not understand, or Cilium somehow manages to route to the root server. Leakage over the public device is ruled out by the root server's Hetzner firewall. Though, it is possible, that this is a bug, that will be fixed and not work anymore, like #133. If so, I was wondering if it would make sense, to use a layer of wireguard peer-to-peer between all nodes, kinda as a unified substrate for cilium. Any clarification on this topic is highly appreciated. |
Cilium uses an overlay network between nodes (vxlan or geneve) by default, maybe you haven't disabled it?
This configuration will work in any way, even without hetzner cloud networks and vswich.
For cilium, this is not necessary, since it already knows how to build tunnels between nodes and does it by default. If encryption is required then cilium supports ipsec (https://docs.cilium.io/en/v1.9/gettingstarted/encryption/). Also I recommend paying attention to latency when connecting vswitch to the cloud: ping from cloud node to dedicated node via public ip:
ping from cloud node to same dedicated node via vswitch:
~0.5ms via public network vs ~46.5ms via private network =(. |
@identw thank you for the hint, you are right, our Cilium uses vxlan as tunnel. That explains why it works. We deploy Istio on top of Cilium, I guess there is no real need for the Cilium encryption for us at the moment. As I understand enabling the Cilium encryption also conflicts with some features of Istio. The ping from a cloud server to the dedicated server via vSwitch is not that bad for us:
Our cloud nodes are hosted in nbg1-dc3 and the dedicated server lives in fsn1-dc15. I guess that would be even better, if we moved the cloud nodes to Falkenstein. FYI we encountered a problem with Cilium and systemd in Debian bullseye, buster is fine: cilium/cilium#14658. |
I mentioned encryption because you wrote about wireguard. Encryption is optional
Not so bad. I tested in the hel1 location (dedicated node from hel1-dc4, cloud node from hel1-dc2).
Thank you interesting. I really also use cilium without kube-proxy, but I have not seen this bug. |
This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs. |
further action occurs |
I saw that someone made a repo (https://github.com/identw/hetzner-cloud-controller-manager) to solve this, has anyone tried it? |
Any updates here? @LKaemmerling are you going to implement support for root server soon? |
@Donatas-L I tried it. It works great with some tidbits. It would need a bit of attention from the community to keep track with the development of the Hetzner Team @LKaemmerling you may also want to have a look here. Maybe you can take this idea ;) |
This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs. |
I also am interested in using bare-metal workers via vSwitch and have it working with calico CNI. Any chance this could become mainlined in the hcloud-cloud-controller-manager? |
If we want to push the european cloud we need to push awesome Hetzner to push itself to grow above itself. This way many open source cloud projects and startups with GDPR and DSGVO compliant ISMS' will able to get founded in Europe. tl;dr yea I'm interested, too. |
I went ahead and rebased the work that @malikkirchner did against
This seems to work next to perfectly with only a couple of transient messages in the cloud-controllers logs such as
...but otherwise load balancer creation works and ignores all nodes that have the I'd file a PR but this really isn't my work, just a few fixes on top of what y'all have already done. Hoping something more legit will make its way into this repo but for now this will have to do. |
@LKaemmerling would you consider reopening this issue as there is a fair bit of support for this feature and quite a bit of hacking that has gone into it already |
@acjohnson thank you for improving on Boris' change. |
Uhm, why is this closed? Currently it does not work. What can I do please? Any step by step instructions how I can provision a LB connected to my 3 root servers? |
It's already full integrated with: https://github.com/syself/cluster-api-provider-hetzner |
Ah, yes. I've read about that CAPI a few days ago already. Thanks mate! |
I'm getting Any Idea how to fix this? |
sounds like you have the wrong provider argument in the deployment... Did you only replaced the image? see: https://github.com/syself/hetzner-cloud-controller-manager/blob/master/deploy/ccm.yaml#L63 |
Well, after removing the "old" ccm, I installed the suggested one with:
Which contains: containers:
- image: quay.io/syself/hetzner-cloud-controller-manager:v1.13.0-0.0.1
name: hcloud-cloud-controller-manager
command:
- "/bin/hetzner-cloud-controller-manager"
- "--cloud-provider=hetzner"
- "--leader-elect=false"
- "--allow-untagged-cloud" Any slack/discord channels available? Don't want to spam this issue here further. |
kubernetes slack workspace channel #hetzner |
Hello,
is it possible to add servers from the Hetzner Robot to the cluster created with the CCM?
I've been using a K3s cluster which I bootstrapped manually and when I tried to install the hcloud CCM the hcloud:// provider was not working for all servers - whether they were Cloud or Robot servers.
Now I've bootstrapped a cluster using kubeadm and followed the instructions and the hcloud:// provider seems to be working, yet I still have my bare-metal servers and before I let them join my cluster and possibly destroy the CCM, I'd rather ask for clarification first.
My expectations would be:
Thank you!
The text was updated successfully, but these errors were encountered: