KubeDns Pods do not share load

[iftachsc]

in our cluster we have 250+- app pods. most of them with lower ttl of 1 minutes generating allot of dns requests.
we had some issues that randomly pods/kubelets cannot resolve dns. we saw that KubeDns pods are working hard. we scaled KubeDns and problem lowered but still occurs.

looking at the KubeDns monitoring it seems that one pod always works hard and the rest almost does nothing. this make sense that the problem will still persist to some extent.

deleting the pod resulted in another new pod that again takes most of the load and not causing load sharing.

@kubernetes/sig-network

[neolit123]

CoreDNS is the preferred DNS server since 1.11-1.12.
please provide details about your setup and follow the issue template.
thank you.

/sig network
/triage support

[iftachsc]

we are using GKE 1.11 so i am not in charge and dont want to change the addons on the cluster.
we have 28 nodes. in our setup as i explained because of low dns ttl there are many requests to kube-dns. we can see clearly the pods work more than a common deployment.
what additional details do you need?

now it looks that 1 works and the others not at all
i dont understand why

[neolit123]

/sig gcp
/remove sig-network

would be still nice to provide the k8s version at least and follow the rest of the issue template:
https://raw.githubusercontent.com/kubernetes/kubernetes/master/.github/ISSUE_TEMPLATE/bug-report.md

thanks!

[neolit123]

/remove-sig network

[BrianChristie]

Have you been able to contact Google Cloud Support for this yet? They would probably be the best resource to start with as I believe cluster DNS resolution falls under the scope of the managed GKE service.

[iftachsc]

Sorry for that. it somehow used the feature request template:
What happened:
we expirience random resolve issues infron of kubedns. some time in the form of resolve failing of gcr
kubelet, gke-ls-prod-od-14c-35g-2bcf01d8-c3v1 Failed to pull image "gcr.io/project-------/-------:5.0.92": rpc error: code = Unknown desc = Error response from daemon: Get https://gcr.io/v2/: dial tcp: lookup gcr.io on 169.254.169.254:53: write udp 172.27.0.145:54958->169.254.169.254:53: write: operation not permitted. looking on kubedns pods they were working hard as we are in an environment with low ttl and all pods will query dns every 1 minute.
We scaled kube-dns deployment to have more replicas but only one pod is high on cpu. the rest hardly a bit.
What you expected to happen:
load to be shared and all kubedns pods to work

How to reproduce it (as minimally and precisely as possible):
no idea.
Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.11.7-gke.6
• Cloud provider or hardware configuration: GCP
• OS (e.g: cat /etc/os-release): COS
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[pswica]

1. Does anything show up when you type kubectl logs kube-dns-<rest-of-pod-name>?
2. How about when you run kubectl get po -n kube-system?
3. Sometimes DNS messes up on Kubernetes if there are issues with the underlying CNI. Can you nslookup/dig these kube-dns servers from other pods on your cluster (e.g. kubectl exec -it <pod> -- nslookup < kube-dns-... )?

[iftachsc]

Update: so after some few deletion the dns pods started to share load na be equal.
regardless we had an issue with Istio ServiceEntries where resolution: DNS was set and it created like 5500 dns queries per second, creating massive load on kubedns that led to some resolve issues from tiem to time in the cluster, including when pulling images from external registry.

removing these solved the issue.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[k8s-ci-robot]

@iftachsc: There are no sig labels on this issue. Please add a sig label by either:

1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
   e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR

2. specifying the label manually: /sig <group-name>
   e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. See the group list.
The <group-suffix> in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.