Plugin [loop] not work with systemd-resolved running

[mritd]

When systemd-resolved is running, nameserver in /etc/resolved.conf default to 127.0.0.53;
The plugin loop detects two DNS query, and finally coredns fails to start.

Environment:

• Ubuntu 18.04.1
• Kubernetes 1.11.2
• CoreDNS 1.2.2

Error log:

docker1.node ➜  kubectl logs coredns-55f86bf584-7sbtj -n kube-system
.:53
2018/09/06 13:02:45 [INFO] CoreDNS-1.2.2
2018/09/06 13:02:45 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2018/09/06 13:02:45 [INFO] plugin/reload: Running configuration MD5 = 86e5222d14b17c8b907970f002198e96
2018/09/06 13:02:45 [FATAL] plugin/loop: Seen "HINFO IN 2050421060481615995.5620656063561519376." more than twice, loop detected

Deploy with deploy.sh

[chrisohaver]

This is working as intended. The loop plugin has detected a forwarding loop, caused by systemd-resolved. If CoreDNS didn't exit, it would loop "forever" on the first upstream query it receives and get OOM killed.

The best fix is to add a flag to kubelet, to let it know that it should use the original resolv.conf....
--resolv-conf=/run/systemd/resolve/resolv.conf, then restart coredns pods

[mritd]

Thanks for your answer, this is a good idea. (I just solved it by stopping systemd-resolved, stupid me 😂).

[miekg]

/plugin: loop /label: question

[avaikararkin]

I am facing same issue:

[root@faas-cent1 ~]# kubectl logs coredns-7f4b9fccc6-6bg7s -n kube-system
.:53
2018/09/28 09:24:50 [INFO] CoreDNS-1.2.2
2018/09/28 09:24:50 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2018/09/28 09:24:50 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
2018/09/28 09:24:56 [FATAL] plugin/loop: Seen "HINFO IN 6010196033322906137.8653621564656081764." more than twice, loop detected

This is on centOS7 & no, my /etc/resolv.conf does not have a 127... entry.
It is this:

[root@faas-cent1 ~]# cat /etc/resolv.conf

Generated by NetworkManager

nameserver 10.148.20.5
[root@faas-cent1 ~]#

[root@faas-cent1 ~]# docker --version
Docker version 18.06.1-ce, build e68fc7a
[root@faas-cent1 ~]#

[root@faas-cent1 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"0ed33881dc4355495f623c6f22e7dd0b7632b7c0", GitTreeState:"clean", BuildDate:"2018-09-27T17:02:38Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
[root@faas-cent1 ~]#

[root@faas-cent1 ~]# uname -a
Linux faas-cent1 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@faas-cent1 ~]#

I don't have a file /run/systemd/resolve/resolv.conf on my system to try the workaround
dnsmasq seems to be running on the system though, would that be causing this issue?

[Asisranjan]

I am getting same error too.
.:53
2018/10/04 12:18:47 [INFO] CoreDNS-1.2.2
2018/10/04 12:18:47 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2018/10/04 12:18:47 [INFO] plugin/reload: Running configuration MD5 = 486384b491cef6cb69c1f57a02087363
2018/10/04 12:18:53 [FATAL] plugin/loop: Seen "HINFO IN 7533478916006617590.6696743068873483726." more than twice, loop detected

[chrisohaver]

This is the loop detection detecting a loop, and exiting. This is the intended behavior, unless of course there is no loop.

If you doubt there is a loop, you may try removing the loop detection (remove loop from the coredns configuration), and then test DNS resolution from pods (i.e. test resolution to external domains from the command line of a pod running in the cluster).

[johnbelamaric]

Seems like the error message needs to be clearer. It should say something like this: Query "HINFO..." seen more than twice. This means a query resolution loop has been detected. CoreDNS will not run until this is resolved, or the loop plugin is removed from the configuration. Leaving a loop in place can cause random DNS resolution failures, crashes, and high CPU utilization. And perhaps even refer to the website to a page that explains some of the common cases we have seen.

…

On Thu, Oct 4, 2018 at 5:36 AM chrisohaver ***@***.***> wrote: This is the loop detection detecting a loop, and exiting. This is the intended behavior, unless of course there is no loop. If you doubt there is a loop, you may try removing the loop detection (remove loop from the coredns configuration), and then test DNS resolution from pods (i.e. test resolution to external domains from the command line of a pod running in the cluster). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2087 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJB4s_kcb5rbvFNoo_KX4W5Qhl6woeYYks5uhgDIgaJpZM4Wc6xv> .

[miekg]

[ Quoting <notifications@github.com> in "Re: [coredns/coredns] Plugin [loop]..." ]

Seems like the error message needs to be clearer. It should say something like this: Query "HINFO..." seen more than twice. This means a query resolution loop has been detected. CoreDNS will not run until this is resolved, or the loop plugin is removed from the configuration. Leaving a loop in place can cause random DNS resolution failures, crashes, and high CPU utilization. And perhaps even refer to the website to a page that explains some of the common cases we have seen.

excellent plan, esp the link to a website

[avaikararkin]

In my case, it seemed to be the problem with IPv6, the VM I had created had IPv6 turned on by default and there was an entry for the same in /etc/resolv. I turned IPv6 off and removed the entries for ::1 and things seem to be working.

[chrisohaver]

Seems like the error message needs to be clearer. It should say something like ...

LOL - i just saw this now, after I submitted a PR for it.

[johnbelamaric]

No problem. @avaikararkin you could add details to the README Troubleshooting section...