[Feature] Customizing CoreDNS ConfigMap with overrides and additional servers

[iwilltry42]

INFO: As per #4397 (comment), the PR was reduced to a single import of *.server overrides

Proposed Changes

Feature: Add CoreDNS Customization Options

Problem:
Before, to customize CoreDNS, one had to edit the default configmap,
which gets re-written on every K3s server restart.

Solution:
Mount an additional coredns-custom configmap into the CoreDNS container
and import overrides and additional server blocks from the included
files.

Types of Changes

New Feature (non-breaking change).

Verification

• build K3s with new feature and use it e.g. via k3d
• deploy a configmap coredns-custom with some modifications, e.g. an additional server:

  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: coredns-custom
    namespace: kube-system
  data:
    example.server: |
      example.org {
        log
        whoami
      }

• run some queries from within the cluster and check the response and coredns logs
  • response expected as per example: https://coredns.io/plugins/whoami/#examples
  • log expected similar to [INFO] 10.42.0.11:52119 - 9047 "A IN example.org. udp 40 false 4096" NOERROR qr,aa,rd 91 0.00010936s

Linked Issues

• More extensibility to CoreDNS configmap #462
• This PR replaces WIP - Coredns user config #743
• [FEATURE] Allow CoreDNS extension with custom configuration k3d-io/k3d#816

User-Facing Change

Allow optional customizations to CoreDNS via `coredns-custom` configmap

Further Comments

Contents of the coredns-custom configmap can include the following keys:

• *.server: defines a new server block
• *.override.start: defines a configuration override that will be imported at the start of the original Corefile (i.e. just before the kubernetes plugin)
• *.override.fallthrough: defines a configuration override that will be imported at in the middle of the original Corefile to define fallthrough actions (i.e. just after consulting the NodeHosts hosts file and before forwarding to `/etc/resolv.conf)
• *.override.end: defines a configuration override that will be imported at the end of the original Corefile (i.e. at the very end of the Corefile)

[iwilltry42]

Rebased upon master to resolve conflict in zz_generated_bindata

[brandond]

@iwilltry42 would you like to backport to 1.22?

[Oats87]

Merging this PR appears to have broken master

[iwilltry42]

@Oats87 , looks like it's (again) the make generate results that differ. I ran that stuff a dozen times but always get slightly different results than CI somehow 🤔
May be related to my Go version 🤔

@brandond that just needs a PR against the release-1.22 branch?

Update: Backport at #4479

[chris13524]

This PR describes support for *.override.* keys that inject configuration into an existing server block:

*.override.start: defines a configuration override that will be imported at the start of the original Corefile (i.e. just before the kubernetes plugin)

However, upon testing, this functionality does not appear to be present and the code only contains an import directive for the *.server keys.

[iwilltry42]

This PR describes support for *.override.* keys that inject configuration into an existing server block:

*.override.start: defines a configuration override that will be imported at the start of the original Corefile (i.e. just before the kubernetes plugin)

However, upon testing, this functionality does not appear to be present and the code only contains an import directive for the *.server keys.

@chris13524 as per this thread, the PR was reduced to a single import of *.server overrides

[lonwern]

Could someone backport this feature to 1.21? since 1.22 removed a lot of old APIs, some of our server have to stay at 1.21.

[ffatghub]

Hi @iwilltry42

I need to add a simple rewrite rule before the kubernetes plugin, like this one in my Corefile

data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        rewrite name foo.bar.test k3d-server-0
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }

I guess coredns-custom is the right tool for the task. So I started extending your example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  test.server: |
    test.org {
      log
      whoami
    }

  test.override.start: |
    test.org {
      log
      whoami
    }

  test.override.fallthrough: |
    test.org {
      log
      whoami
    }

  test.override.end: |
    test.org {
      log
      whoami
    }

dig test.org shows the whoami plugin is invoked and a log entry is written by coredns

I would have expected multiple import statements in the coredns main config map, however only the .end (I guess?) is present

data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    import /etc/coredns/custom/*.server

I thought *.override.start was the right "hook" but if I put a rewrite using *.override.start nothing happens (CoreDNS correcly reloads the custom map)

Am I missing something?

thanks a lot
fra

[iwilltry42]

Hi @ffatghub , please see #4397 (comment)

[andrevtg]

Should this support wildcards? Not working for me in k3d.

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  localdomain.server: |
    localdomain {
        hosts {
            192.168.65.254 *.localdomain
            192.168.65.254 whoami.localdomain xxx.localdomain
            ttl 60
            fallthrough
        }
    }

In k3d I have managed to make it work with rewrite instead:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  rewrite.override: |
    rewrite name regex (.*)\.localdomain host.k3d.internal