[Enhancement] Edit CoreDNS ConfigMap on disk for more reliability

[iwilltry42]

Goal

• add host entries to the existing NodeHosts section of the CoreDNS configmap
  • host.k3d.internal -> docker host reference
  • k3d nodes (K3s node containers) -> also managed by the K3s node controller
  • k3d helper nodes (serverlb, registry, etc.)
  • other containers in the same docker network

Current Way

• basically kubectl patch configmap ...
• problems: has to wait for cluster to be ready
  • coredns configmap must be present
  • API Server must be responsive
• impact: fails to create cluster in some cases, even with lots of retries

New Way

• wait for coredns.yaml to be written to disk
• read coredns.yaml from disk, modify the configmap included to add our entries to NodeHosts
• let the K3s controller re-deploy the configmap (very fast)
• let CoreDNS auto-reload the config (up to 30s due to the default reload interval)
• benefits: more reliable, no retries needed, no need to wait for other parts of K3s to be up and running

Future

This is still hacky, so we're hoping for an alternative solution soon, e.g. similar to this: https://github.com/k3s-io/k3s/pull/743/files

References

fixes #786
potentially fixes #805