New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
coredns pods connect to coredns service timeout #4674
Comments
What is the content of /etc/resolv.conf on the host |
@woopstar The content is my local dns server's ip such as "nameserver 10.19.8.10", this dns server is active. |
We are facing the same issue, when we have the 2 node replica of coredns pods. One of the coredns pods throws the same error. |
I was using microk8s with this warning WARNING: IPtables FORWARD policy is DROP. Consider enabling traffic forwarding with: sudo iptables -P FORWARD ACCEPT fixing it worked for me |
You saved my life, bro! I'm trying to setup a k8s cluster on several Raspberry Pis, facing the same problem and tried everything I can but finally it's the iptables problem, get it fixed with the forward policy set to accept. |
Following @jeremyrickard 's comments, another thing to notice is that Docker will set the iptable rules, making FORWARD DROP. So if you are using systemd to manage docker, remember to add the command 'iptables -w -P FORWARD ACCEPT' as a post exec start script in docker.service file. Hope this helps. |
the |
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Just for someone looking into the same issue. I am running a cluster with 4 raspberry pi 4 and I encounter this issue in each node. @jesusmariajurado was right about the solution but it was not enough for me. There were other rules that were preventing my pods to reach out external ips. Flushing all rules, deleting alll chains, and accept all work form me, as it is said in the link: https://www.digitalocean.com/community/tutorials/how-to-list-and-delete-iptables-firewall-rules What I am observing now is this: kubernetes/kubernetes#82361 running on 1.16.2. I am gonna try to update k8s to 1.17 as it seems solved Might be worthy to note as well (for others having problems with coredns such as connections timeouts or not being able to resolve names) that the nameserver defined in the /etc/resolv.conf must be an upstream server or else will create loopbacks: https://coredns.io/plugins/loop/#troubleshooting |
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
Rotten issues close after 30d of inactivity. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Environment:
Cloud provider or hardware configuration:
**OS **
Linux 3.10.0-693.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
ansible --version
):ansible 2.7.10
Kubespray version (commit) (
git rev-parse --short HEAD
):tag 2.9.0
v1.13.5
Network plugin used:
calico
I ues kubespray v1.13.5 to deploy a k8s service,but error occurs .Bellow is the version of coredns and error log.
Version of CoreDNS : v1.4.0 (also test with coredns v1.5.0)
Corefile:
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream /etc/resolv.conf
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
Logs:
[root@k8s-node-1 ~]# kubectl logs -f -n kube-system coredns-9d85fb698-tnrgn
.:53
2019-04-29T12:26:42.180Z [INFO] plugin/reload: Running configuration MD5 = 1335ba7188be742fe37cd05805faa0fa
2019-04-29T12:26:42.180Z [INFO] CoreDNS-1.5.0
2019-04-29T12:26:42.180Z [INFO] linux/amd64, go1.12.2, e3f9a80
CoreDNS-1.5.0
linux/amd64, go1.12.2, e3f9a80
2019-04-29T12:26:48.181Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:51809->10.233.0.3:53: i/o timeout
2019-04-29T12:26:51.181Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:52463->10.233.0.3:53: i/o timeout
2019-04-29T12:26:52.181Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:44654->10.233.0.3:53: i/o timeout
2019-04-29T12:26:53.181Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:35028->10.233.0.3:53: i/o timeout
2019-04-29T12:26:56.181Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:44331->10.233.0.3:53: i/o timeout
2019-04-29T12:26:59.182Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:38640->10.233.0.3:53: i/o timeout
2019-04-29T12:27:02.182Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:57424->10.233.0.3:53: i/o timeout
2019-04-29T12:27:05.182Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:56166->10.233.0.3:53: i/o timeout
2019-04-29T12:27:08.182Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:59509->10.233.0.3:53: i/o timeout
2019-04-29T12:27:11.183Z [ERROR] plugin/errors: 2 8373768935828175380.8715076686105595443. HINFO: read udp 10.233.113.56:56157->10.233.0.3:53: i/o timeout
The 10.233.0.3:53 is coredns service and 10.233.113.56 coredns pods .
When I configure /etc/resolv.conf on host which the coredns pod is running on, the coredns pods is running, but keeping content of the /etc/resolv.conf is empty, the pods will not run, and the log in coredns pods tells me timeout.
From the iptables rule output, the coredns service has no endpoints.
The 'timeout’ and 'no endpoints' seems like a deadlock.
The text was updated successfully, but these errors were encountered: