coredns issues on v1.13

[moltar]

Not sure if this is kind related, or a general Kubernetes issue.

But I'll open it here anyways. If you feel that it is misplaced, please let me know.

Consistently getting failure on CoreDNS pods.

kube-system        pod/coredns-869f847d58-wsp5d           0/1     CrashLoopBackOff   5          5d4h

.:53
2019/03/30 03:22:37 [INFO] CoreDNS-1.2.2
2019/03/30 03:22:37 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2019/03/30 03:22:37 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
2019/03/30 03:22:43 [FATAL] plugin/loop: Seen "HINFO IN 2541451446677734580.3252529887689954137." more than twice, loop detected

Thanks.

[moltar]

Seems related: coredns/coredns#2087

[moltar]

Just tested, and the following fix does indeed work:

coredns/coredns#2087 (comment)

[nhoughto]

Can't repro?

MacBook-Pro:kind nickhoughton$ docker run -it --privileged bsycorp/kind:latest-1.13
2019-03-31 01:23:13,113 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2019-03-31 01:23:13,133 INFO RPC interface 'supervisor' initialized
2019-03-31 01:23:13,134 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-03-31 01:23:13,134 INFO supervisord started with pid 1
2019-03-31 01:23:14,138 INFO spawned: 'start' with pid 7
Clean up..
Setting up networking..
Extracting cache..
2019-03-31 01:23:15,175 INFO success: start entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-03-31 01:23:22,477 INFO spawned: 'dockerd' with pid 17
2019-03-31 01:23:23,556 INFO success: dockerd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
dockerd: started
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 10
Server Version: 18.09.3
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.125-linuxkit
Operating System: Alpine Linux v3.9 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.855GiB
Name: 3d19ebb8ffd8
ID: WVYC:VGZX:HAMI:WOZD:JRQO:U6NW:RZFQ:NDVN:6PBR:JQSH:ZDX6:JMWG
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Docker ready
Logging into docker-registry  with user 
no docker-registry/-credentials supplied
Starting config server..
2019-03-31 01:23:27,527 INFO spawned: 'config-serve' with pid 200
2019-03-31 01:23:28,779 INFO success: config-serve entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
config-serve: started
Starting Kubernetes..
2019-03-31 01:23:29,032 INFO spawned: 'kubelet' with pid 284
2019-03-31 01:23:30,036 INFO success: kubelet entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
kubelet: started
Checking startup status..
Checking startup status..
startup successful
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-869f847d58-wsp5d           1/1     Running   0          6d2h
kube-system   etcd-minikube                      1/1     Running   0          6d2h
kube-system   kube-addon-manager-minikube        1/1     Running   0          6d2h
kube-system   kube-apiserver-minikube            1/1     Running   0          6d2h
kube-system   kube-controller-manager-minikube   1/1     Running   0          6d2h
kube-system   kube-proxy-qr6pp                   1/1     Running   0          6d2h
kube-system   kube-scheduler-minikube            1/1     Running   0          6d2h
Kubernetes ready
2019-03-31 01:23:53,959 INFO exited: start (exit status 0; expected)

[moltar]

Hmmm ya, it only seems to happen with a specific setup. I'll try to create a repro manifest. Don't close the issue yet, please.

[nhoughto]

Were you able to repro?

[moltar]

Not at this point. Let's assume this is fixed for now and close the issue. Thanks!

[gerardo]

This happens specifically with Docker Compose. Someone else has encountered this problem as well siderolabs/talos#575

[nhoughto]

Do you have a repro to test against? Seems like an easy change if can test it..

[gerardo]

@nhoughto sure, here it is. With this docker-compose file https://gist.github.com/gerardo/b4fe5effbf5c4490c74e53f19f750b95:

$ docker-compose run -u root --entrypoint bash kubectl
Creating kubernetes_1 ... done

root@182238fc26a3:$ mkdir -p $HOME/.kube/

root@182238fc26a3:$ wget http://kubernetes:10080/config
--2019-05-31 01:01:27--  http://kubernetes:10080/config
Resolving kubernetes (kubernetes)... 172.22.0.2
Connecting to kubernetes (kubernetes)|172.22.0.2|:10080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5435 (5.3K) [application/octet-stream]
Saving to: 'config'

config                       100%[==============================================>]   5.31K  --.-KB/s    in 0.001s

2019-05-31 01:01:27 (5.90 MB/s) - 'config' saved [5435/5435]

root@182238fc26a3:$ cp config $HOME/.kube/config

root@182238fc26a3:$ kubectl config set clusters.kind.server https://kubernetes:8443
Property "clusters.kind.server" set.

root@182238fc26a3:$ kubectl get -n kube-system pods
NAME                               READY   STATUS             RESTARTS   AGE
coredns-fb8b8dccf-hkwrv            0/1     CrashLoopBackOff   1          27d
coredns-fb8b8dccf-q5r54            0/1     CrashLoopBackOff   1          27d
etcd-minikube                      1/1     Running            0          27d
kube-addon-manager-minikube        1/1     Running            0          27d
kube-apiserver-minikube            1/1     Running            0          27d
kube-controller-manager-minikube   1/1     Running            0          27d
kube-proxy-mn786                   1/1     Running            0          27d
kube-scheduler-minikube            1/1     Running            0          27d

root@182238fc26a3:$

[gerardo]

Hey @nhoughto have you found the problem? Is there anything I can do to help?

[nhoughto]

Your repro is great, just need to sit down and do it, basically need the build.sh process to edit the coredns config and remove loop as per the linked issue. You can try and PR in if you like, otherwise hoping to get to it this week.