More extensibility to CoreDNS configmap

[mlebrun]

Is your feature request related to a problem? Please describe.
We have a desire to point requests from a specific domain (that doesn't line up with the current cluster domain schema) to the ingress in order to route requests from a web head pod to another pod on the same (single) node. For instance:

(webhead) web.dev.be.lan has an env var pointing to (other pod on same node) search.dev.be.lan.

We currently modify the rendered/written manifest at

/var/lib/rancher/k3s/server/manifests/coredns.yaml

To resemble:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        rewrite name regex [a-zA-Z.]+\.dev0\.be\.lan traefik.kube-system.svc.cluster.local <-- added
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

The problem is, on k3s server start, this file gets overwritten:
https://github.com/rancher/k3s/blob/master/pkg/server/server.go#L133

Describe the solution you'd like
Ideally, there would be a way to augment the config map in a clean way so that when the file is updated/overwritten on server start, the changes persist.

Describe alternatives you've considered
Another option I considered was copying the CoreDNS manifest into our repo, adding the line, and specifying --no-deploy coredns but I didn't realize there were template variables in the file: %{CLUSTER_DOMAIN}% and %{CLUSTER_DNS}% so I'm reluctant to go the route as cluster DNS is very likely to not be consistent between environments/nodes.

I'm not sure if there's another route I'm missing to either overwrite the config or reference it in another spot to survive restarts, but I'm definitely open to testing any solutions and experimenting a bit. Thank you for all the hard work on this project, it's great having a lightweight solution like k3s!!

[mlebrun]

Maybe it would be possible to allow some usage of https://coredns.io/plugins/import/ with a CLI arg?

[jait]

Or maybe the system could allow defining a custom name for the CoreDNS ConfigMap? This way the default configmap could stay in the system and if custom configs are needed, the admin could just create a new ConfigMap and point the configuration to load the Corefile from there?

[jait]

Turns out there's (a bit hacky) way to have custom CoreDNS configs: the manifests in /var/lib/rancher/k3s/server/manifests are applied in order sorted by filename. So, to override the default CoreDNS config, one can simply create a new file e.g. with a name d_coredns-config.yaml and there you can override the coredns ConfigMap.

Question for Rancher Labs: can we rely on this behaviour and what is their vision of the optimal way to do this?

[mlebrun]

I was going to say, that might not leave us in much of a better spot, but since it's just the CoreDNS ConfigMap, I suppose that could work. At least for the time being.

[erikwilson]

We could probably use the import plugin to achieve this, am doing something similar with hosts on k3s where the current coredns configmap looks like the following:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          reload 1s
          fallthrough
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
  NodeHosts: |
    10.135.135.100 k3s

[erikwilson]

But in general manifest stuff could probably be improved, would depend on a properly named patch.

[erikwilson]

The main issue with import is it doesn't really support graceful reloading so would require restarting coredns or updating Corefile.

[erikwilson]

Related: coredns/coredns/issues/2633

[mlebrun]

@jait I don't believe it's reliable behavior. I've got an Ansible task that exports the config itself from the k8s resource via kubectl and writes it to override_coredns.yaml, then deletes the coredns configmap and recreates it. Seems to work after it runs but while the file survives restarts, the functionality doesn't. I might be missing something, but but doesn't seem to work at least reliably.

[mlebrun]

@erikwilson have you made any progress on your PR here? erikwilson/coredns#1

[flxs]

Are there any news on this? The workaround mentioned above seems to work (for me) right now, but apparently it's a rather unstable situation.

[erikwilson]

I opened a PR with coredns, please give it a thumbs up if it looks good: coredns/coredns#3068, might help to move it along.

[mlebrun]

@erikwilson looks like your PR was merged, what do we need to do to get that work leveraged here? I'm willing to help with whatever I can.

[brokenjacobs]

Another note on this. Even if you override the configmap, when you join a node to the cluster it overwrites the configmap with a node list. Open to any suggestions here.

[knvpk]

anyone has better solution.

[bbhenry]

We are having the same issue here. Does anyone know what's causing the coredns manifest file to override on restart?

[brandond]

All packaged components are re-deployed whenever a server node is restarted. If you want to replace coredns or any other component with your own, you can --disable it.

[boeboe]

@bradtopol .... I do not want to disable the coredns plugin, I just want to be able to save changes made to the configmap for multi cluster scenario's. What is your advise to achieve this?

[febus982]

Could it be simpler if instead of generating a single ConfigMap containing both Corefile and NodeHosts, generating two separate ConfigMap manifests and mount them as separate volumes.

With this approach, it could be possible to adopt either of these solutions:

• allow overriding the ConfigMap containing Corefile while continuing to update the NodeHosts one
• allow disabling CoreDNS and allowing keeping the NodeHosts ConfigMap and the automatic update (it would be up to the k3s administrator manage the reload of any custom CoreDNS implementation)

[johgoe]

Maybe it's possible to do it similar to Azure Kubernetes? The coredns configuration on Azure looks like

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        ready
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
        import custom/*.override
    }
    import custom/*.server

and it mounts a config map with fixed name coredns-custom to folder custom. Depending on the files in the configmap *.override or *.server you are able to do a lot of customizings.

Some examples are explained here:
https://docs.microsoft.com/de-de/azure/aks/coredns-custom

[Blackclaws]

This feature would be great as we have an internal dns server set up to handle internal traffic and we don't want to foward all requests to this server. It seems a part of the feature is in the next release?

[bguzman-3pillar]

Validations steps:
Using version: v1.23.1-rc1+k3s1
Create 1 server node and 1 agent node.
Deploy a coredns-custom, run commands and look the output.

Once server is ready, deployt this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  example.server: |
    example.org {
      log
      whoami
      }

$ kubectl get all -A -o wide

.
.
NAMESPACE     NAME                                         READY   STATUS      RESTARTS   AGE   IP          NODE               NOMINATED NODE   READINESS GATES
kube-system   pod/coredns-84c56f7bfb-qcl4q                 1/1     Running     0          24m   10.42.0.4   ip-172-31-37-181   <none>           <none>
.
.

• CoreDNS response expected as per example was correct:

; <<>> DiG 9.16.1-Ubuntu <<>> @10.42.0.4 example.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53498
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 9aea3393a5e07e6c (echoed)
;; QUESTION SECTION:
;example.org.			IN	A

;; ADDITIONAL SECTION:
example.org.		0	IN	A	10.42.0.1
_udp.example.org.	0	IN	SRV	0 0 58810 .

• For logs, this warning was displayed:
  kubectl logs -n kube-system coredns-84c56f7bfb-qcl4q

[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server
[WARNING] No files matching import glob pattern: /etc/coredns/custom/*.server

[m-bers]

I see import /etc/coredns/custom/*.server is in the file /var/lib/rancher/k3s/server/manifests/coredns.yml but I would like to use coredns with the file plugin in order to create a wildcard CNAME record pointing to my ingress controller. Is this possible?

Based on @johgoe's comment and the corresponding documentation linked in this thread, the Azure implementation supports all coredns plugins via import custom/*.override but I didn't see that line in the coredns.yml in k3s's manifests dir.

[michael-riha]

@m-bers maybe this is what you are looking for?

k3s/manifests/coredns.yaml

Line 186 in 1d4f995

- name: custom-config-volume