fix: preserve coredns config during cluster restart using the coredns-custom configmap (!)

[cjc7373]

What

fixes #1221

Why

Since coredns configmap will be reverted to original state during k3s startup process, this PR utilizes coredns-custom configmap (introduced in k3s-io/k3s#4397) to persistent custom coredns configs (e.g. the host.k3d.internal record).

Since coredns's host plugin can only be used once per server block, I have to use the file plugin.

A go template is used to render the configmap. I don't know if there's a better way.

Implications

Not I'm aware of.

[iwilltry42]

Thanks for this @cjc7373 !
FWIW, an alternative using the hosts plugin would be adding another server block using a *.server override, but that would also mean using a different zone and having a suffix, which would be kind of bad.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  k3d.internal.server: |
    k3d.internal:53 {
        errors
        health
        ready
        hosts /etc/coredns/custom/k3dhosts {
            ttl 60
            reload 15s
            fallthrough
        }
        cache 30
    }
  k3dhosts: |
    1.2.3.4 foobar.k3d.internal
    2.1.2.1 foobar.foo.local

[iwilltry42]

Tested and confirmed that it's working as expected ✔️
Thanks again!

[iwilltry42]

@cjc7373 I feel like there's no perfect way of achieving this.
The options we have of customizing coredns using the coredns-custom configmap are limited (I was aiming for more customizability, but couldn't get it into K3s - k3s-io/k3s#4397).

Anyway, also the coredns-custom configmap is something that users may want to use themselves, so we have to be careful with it.

I'm thinking of some other ways:

1. dropping a tiny daemon into the K3s containers that rewrites the CoreDNS configmap once it gets written by K3s
2. always disabling CoreDNS and writing it ourselves (risky, as K3s does some config and templating magic - which we could hijack as well)
3. dropping a tiny DNS server binary into the K3s containers and serving that - included via an override in CoreDNS (forward plugin)
4. Finally, after so many applications that we had for it, resorting to a k3d daemon (either on the host system or in the form of the k3d-tools container) that takes care of "continuous monitoring and maintenance" of K3s node containers (I personally still dislike the idea of having any sort of host-local k3d daemon)

WDYT?