forward plugin "sequential" policy is not always respected

[mmiller1]

We are using the forward plugin on coredns 1.1.3 to resolve external dns names, we set the policy to sequential as to only hit the primary dns server provided, in our environment this is a load balanced endpoint that will always respond much faster than our alternative servers. See our configuration below:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |-
    .:53 {
        cache 30
        debug
        errors
        forward . /etc/resolv.conf {
          policy sequential
          max_fails 2
          health_check 1s
        }
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        loadbalance round_robin
        log
        prometheus 0.0.0.0:9153
        reload
    }

our resolv.conf file looks as follows:

$ kubectl exec -ti coredns-65fc98d5c6-9r6rn cat /etc/resolv.conf -n kube-system
nameserver 10.130.150.10
nameserver 10.130.170.240
nameserver 10.130.170.241
search domain1 domain2 domain3

while on the nodes that the coredns pods are running on, the following command will show a good amount of dns traffic being routed to the alternative dns servers.

tcpdump -i bond0 port 53 and not host 10.130.150.10

This is also evident in the prometheus metrics.

sum(coredns_forward_request_count_total) by (to)
{to="10.130.150.10:53"} | 839073
{to="10.130.170.240:53"} | 26870
{to="10.130.170.241:53"} | 1529

There is nothing in the coredns logs that seem to indicate why the alternative servers are receiving traffic, any help understanding / resolving this problem is appreciated.

[miekg]

[ Quoting <notifications@github.com> in "[coredns/coredns] forward plugin "s..." ]

``` sum(coredns_forward_request_count_total) by (to) {to="10.130.150.10:53"} | 839073 {to="10.130.170.240:53"} | 26870 {to="10.130.170.241:53"} | 1529 ```

For a reason the first query fails and then the second (and third) server is tried. logging here would make sense if we can do `log.Sometimes` to not utterly spam the logs.

[mmiller1]

It would be awesome to see some additional logging in this area, as far as I can tell, we never had this issue with kubedns.

[miekg]

intent to close this soon