Feat change of cluster and service cidr

[tschuering]

@mysticaltech

Problem

You cannot change the cluster_ipv4_cidr and service_ipv4_cidr really. I tested this with Cilium and if changed those values to...

cluster_ipv4_cidr = "10.48.0.0/12"
service_ipv4_cidr = "10.64.0.0/12"

The ClusterIP of "kubernetes" service in default namespace was well referencing to 10.64.0.1 but the rest of the services were still running in 10.43.0.0/16 CIDR. To permanent set those values, you need to make sure that these values are fixed set in the config.yaml of k3s. This was well the case for the initial config create in init.tf for the first control plane node, but not the config update later on all control plane nodes. Also what it is missing for that feature to get really this working is setting the cluster-dns value which defines the IPv4 address of the core-dns service, which is always set fixed by k3s. Here a screenshot which gives you an overview of those values for "k3s server" command:

Solution and therefore PR

1. Set cluster-cidr and service-cidr permanent in control_planes.tf
2. Add new config variable for kube-hetzner called cluster_dns_ipv4 for setting the IPv4 address of the core-dns service, which needs to sit in the service-cidr
3. Added variable and default value of k3s in variables.tf
4. Added some explanation in kube.tf.example and also hints that you should never change those values after initialization of a cluster, otherwise this would lead probably to serious consequences. I am not 100% sure about the DNS part, but I guess it is better to let this stay there too.

Proof it works

[mysticaltech]

LGTM, good improvements

[mysticaltech]

@Silvest89 @M4t7e Could you please validate this too please 🙏

[M4t7e]

LGTM!

cluster-cidr and service-cidr should actually be added to control_planes.tf through #902. Thx for fixing that and adding the DNS setting! See also my old comment: #902 (comment)

[Silvest89]

LGTM

[mysticaltech]

@tschuering Awesome contributions. Thank you for that. Both PRs have been merged and released in v2.9.2! 🚀