Enable server certificate bootstrap for the Kubelet in the KinD cluster

> Instead of self signing a serving certificate, the Kubelet will request a
> certificate from the 'certificates.k8s.io' API. This requires an approver to
> approve the certificate signing requests (CSR).
https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

The CSRs are approved after the KinD cluster is created.

With this setting, the Kubelet's certificate is issued by the default CA
(kube-root-ca.crt) that Prometheus trusts in the current configuration, and the
certificate contains a SAN IP entry for the node IP so that Prometheus can
successfully verify the certificate.

openssl s_client -connect 172.18.0.2:10250 -showcerts </dev/null 2>/dev/null | openssl x509 -text | grep -A 1 'Subject Alternative Name:'
  X509v3 Subject Alternative Name:
    DNS:gardener-local-ha-multi-zone-worker2, IP Address:172.18.0.2

So, at this state of the PR, the local setup works, and it would also
work in GKE managed clusters, where the Kubelet's certificate is issued by
kube-root-ca.crt.

However, it would not work in seeds that are Gardener shoots, because
currently the Kubelet's certificates are issued by a separate CA,
  ca-kubelet-...
which is available in the control plane of the seed, but not in the seed itself.
Maybe the Gardenlet in the soil could copy this CA to the kube-system namespace
of its shoot, and the Gardenlet in the seed, if it finds a ca-kubelet secret in
the kube-system namespace, could copy it to the garden namespace. A conditional
Prometheus volume could be used to mount this ca-kubelet to Prometheus if it
exists (managed seed) or mount kube-root-ca.crt otherwise. This condition
handling has to happen in Gardener because Prometheus expects a single CA in its
configuration:
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config

example/gardener-local/kind/cluster/templates/_kubeadm_config_patches.tpl

@@ -2,7 +2,7 @@
 - |
   kind: ClusterConfiguration
   apiServer:
-{{- if .Values.gardener.apiserverRelay.deployed }}  
+{{- if .Values.gardener.apiserverRelay.deployed }}
     certSANs:
       - localhost
       - 127.0.0.1
@@ -29,4 +29,5 @@
   serializeImagePulls: false
   registryPullQPS: 10
   registryBurst: 20
+  serverTLSBootstrap: true
 {{- end -}}

hack/kind-up.sh

@@ -394,3 +394,27 @@ if [ $felix_config_found -eq 0 ]; then
   echo "Error: FelixConfiguration 'default' not found or patch failed after $max_retries attempts."
   exit 1
 fi
+
+# Auto approve Kubelet Serving Certificate Signing Requests: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#kubelet-serving-certs
+# such that the Kubelet in the KinD cluster uses a certificate signed by the cluster's CA: 'kubernetes'
+# instead of a self-signed certificate generated by the Kubelet itself.
+# The CSR is created with some delay, so for each node, wait for the CSR to be created.
+# There can be multiple CSRs for a node, so approve all of them.
+echo "Approving Kubelet Serving Certificate Signing Requests..."
+for node in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
+  max_retries=600
+  for ((i = 0; i < max_retries; i++)); do
+    csr_names=$(kubectl get csr -o json | jq -r --arg node "$node" '
+      .items[] | select(.status=={} and
+                        .spec.signerName == "kubernetes.io/kubelet-serving" and
+                        .spec.username == "system:node:"+$node) | .metadata.name')
+    if [ -n "$csr_names" ]; then
+      for csr_name in $csr_names; do
+        kubectl certificate approve "$csr_name"
+      done
+      break
+    fi
+    sleep 1
+  done
+done
+echo "Kubelet Serving Certificate Signing Requests approved."