create cluster load balancer with terraform

[schlichtanders]

The official hcloud-cloud-controller-manager documentation states that you can also predefine your load balancer with terraform.

Advantages:

• load balancer is created slightly faster
• terraform destroy destroys everything - no need for cleanup any longer cleanup is no longer needed for the load balancer

This pull-requests also includes a slight update to the restoration readme (restoration was again failing for me on recent updates - I updated the documentation to reflect the extra changes to make it more stable)

EDIT: I tested the creation and everything runs through without problems

[schlichtanders]

I need to correct my statement:

• while terraform destroy completes successfully, the firewall and only the network is not deleted - so the cleanup is still needed

[mysticaltech]

I need to correct my statement:

• while terraform destroy completes successfully, the firewall and only the network is not deleted - so the cleanup is still needed

Yes, that would make sense, they do stick around because of the autoscaled nodes maybe.

[mysticaltech]

@schlichtanders Thanks for this, I remember us talking about it. It's awesome. However can you confirm that both traefik and nginx, the ingress controllers, will take that already created lb and apply the custom settings they have, like all the hetzner lb annotations (see kube.tf.example traefik and nginx helm values), including for the hostname?

[schlichtanders]

Dear @mysticaltech I can confirm for traefik, that everythings runs without problems - I have deployed my cluster using this branch.
Also Autoscaling works, really everything works so far I can see. I haven't tested nginx, but because it uses the same hcloud go routines under the hood, it should indeed do the very same.

[mysticaltech]

Ok wonderful, good to hear, I will try nginx myself.

@kube-hetzner/core FYI Big positive evolution in this PR!

[mysticaltech]

🥳