Proper way to rewrite custom DNS entries in Azure Kubernetes Service (AKS)

[sgerace]

We're trying to solve a specific problem in our Azure Kubernetes Service (AKS) cluster, that I'm hoping we can get some help from this community on. We're trying to utilize a public domain name when communicating between services in the cluster (e.g., example-cluster.net instead of default.svc.cluster.local) so that we can obtain a valid certificate from a CA authority and utilize this certificate in both internal and external communication (basically the same issue described in #2347).

Where our problem differs from the previous issue is that we are trying to apply these configuration changes to a Kubernetes cluster running in Azure, which has specific limitations on what you can and can not alter as far as the CoreDNS configuration is concerned (see https://docs.microsoft.com/en-us/azure/aks/coredns-custom). Interestingly, the Microsoft docs article mentions the ability to configure a custom ConfigMap that allows for use of DNS rewriting (see https://docs.microsoft.com/en-us/azure/aks/coredns-custom#rewrite-dns), however, using the provided example does not work properly in our cluster. I have opened MicrosoftDocs/azure-docs#86902) to address the issue that the documented example does not work, but I now have a more fundamental question around why it doesn't work, and the "proper" way of thinking about this problem.

Here is the high-level details. Within AKS, we are unable to modify the default coredns ConfigMap, which is set as the following:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        ready
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
        import custom/*.override
    }
    import custom/*.server

All working examples of this approach shows modifying this directly to include the rewrite directive before the kubernetes cluster.local in-addr.arpa ip6.arpa. However, in AKS we need to create a separate *.server ConfigMap under the coredns-custom entry (as described in the Microsoft documentation). If we utilize the example provided, i.e., something like:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  example-cluster.server: | # you may select any name here, but it must end with the .server file extension
    example.com:53 {
      log
      errors
      rewrite stop {
        name regex (.*)\.example-cluster\.net\.$ {1}.default.svc.cluster.local
        answer name (.*)\.default\.svc\.cluster\.local\.$ {1}.example-cluster\.net
      }
      forward . /etc/resolv.conf # you can redirect this to a specific DNS server such as 10.0.0.10, but that server must be able to resolve the rewritten domain name
    }

We do not see the correct behavior. Requests are not routed properly and if we examine the logs from the coredns pods, we see the following:

[INFO] 10.0.20.18:56053 - 58809 "A IN test.example.com. udp 60 false 512" NXDOMAIN qr,rd,ra 121 0.003595002s
[INFO] 10.0.20.18:56053 - 59326 "AAAA IN test.example.com. udp 60 false 512" NXDOMAIN qr,rd,ra 121 0.004423049s

However, if we basically reproduce the entire default config map and add the rewrite directive:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  example-cluster.server: |
    example-cluster.net:53 {
      log
      errors
      ready
      health
      rewrite stop {
        name regex (.*)\.example-cluster\.net\.$ {1}.default.svc.cluster.local
        answer name (.*)\.default\.svc\.cluster\.local\.$ {1}.example-cluster\.net
      }
      kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
      }
      prometheus :9153
      forward . /etc/resolv.conf
      cache 30
      loop
      reload
      loadbalance
    }

I think this behavior makes sense to me, CoreDNS does not "fallthrough" by default and for some reason the resolution requires the kubernetes statement to properly resolve. However, I'm a bit scared of this solution because what if AKS decides to change their default behavior? Our custom DNS is essentially overwriting ALL relevant internal traffic through this new entry and so there is a possibility that it could break as a result of that kind of a change (though I'm not sure how often that really changes).

As such, the question is: what is the "proper" way to implement this sort of configuration when we are unable to modify the original coredns ConfigMap.

[chrisohaver]

Just want to say that I find Azure's Corefile customization restrictions frustrating from a support perspective. I don't administer an AKS cluster, so I can't speak from experience from an administrative perspective. I know this isn't helpful.

[chrisohaver]

The execution order of plugins is not governed by the order of the plugins in the Corefile.

Plugins are executed in a predetermined order based on the order in plugin.cfg cemented at compile time. So, a rewrite can be added via a custom/*.override, which is imported into the default plugin serve block. Even though those plugins get inserted into the end of the plugin block, the execution order of the plugins is still as per plugin.cfg, and rewrite gets executed before kubernetes.

Therefore (unless I'm missing something) your proposed solution isn't really necessary. Negatively, it results in keeping two separate instances of the Kubernetes API informer in memory, which at scale would double the memory consumption of CoreDNS.

[sgerace]

Thanks @chrisohaver. Are you therefore suggesting that something like this — which is now an *.override instead of an *.server — would be the correct approach?

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  example-cluster.override: |
    rewrite stop {
      name regex (.*)\.example-cluster\.net\.$ {1}.default.svc.cluster.local
      answer name (.*)\.default\.svc\.cluster\.local\.$ {1}.example-cluster\.net
    }

I'm asking about the syntax as well, I'm assuming that since it is included within the root block it only needs the rewrite portion of the configuration.

[chrisohaver]

Yes, thats what I am suggesting. Sans one typo in the answer rewrite I think...

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  example-cluster.override: |
    rewrite stop {
      name regex (.*)\.example-cluster\.net\.$ {1}.default.svc.cluster.local.
      answer name (.*)\.default\.svc\.cluster\.local\.$ {1}.example-cluster.net.
    }

[sgerace]

Awesome, makes total sense now (much better solution than what is provided in Microsoft docs). I'll give this a try in a bit and let you know how it goes.

[sgerace]

This seems to work perfectly, thanks for the quick response and help @chrisohaver!