Add troubleshooting section to kube-dns readme #1449

MrHohn · 2016-10-12T22:57:22Z

This PR added some generic troubleshooting information for kube-dns addon. The addition is pasted below.

Troubleshooting tips.

If above nslookup command does not work, here are some hints:

Check the local DNS configuration first

Take a look inside the resolv.conf file. (See "Inheriting DNS from the node" and "Known issues" sections for more information)

cat /etc/resolv.conf

You should see the search path and nameserver are set up like below: (search path may vary for different cloud providers)

search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal
nameserver 10.0.0.10
options ndots:5

DNS known issues

If you are using Alpine(lower than 3.2) as base image, dns may not work properly because Alpine broke dns search paths but fixed it in some later release. Please take a look at here for more detail.

Quick diagnosis

If below appears, usually means something is wrong with kube-dns addon or Services

/ # nslookup kubernetes
Server:    10.0.0.10
Address 1: 10.0.0.10

nslookup: can't resolve 'kubernetes'

or

/ # nslookup kubernetes
Server:    10.0.0.10
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'kubernetes'

Let's continue to debug it.

Is dns pod running?

kubectl get pods command could tell. (namespace may be "default" or others if you manually deploy it)

kubectl get pods --namespace=kube-system -l k8s-app=kube-dns

You should see something like:

NAME                                                       READY     STATUS    RESTARTS   AGE
...
kube-dns-v19-ezo1y                                         3/3       Running   0           1h
...

If no pod is running or failed/completed, then maybe because this dns addon would not be deployed by default in your current environment and you have not deployed it manually.

Otherwise, please try next hint.

Is dns pod working properly?

Use kubectl logs command for detective work.

kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c healthz

See if you could find any suspicious log. If not, please continue.

Is dns service up?

Varify through kubectl get service command.

kubectl get svc --namespace=kube-system

You should see something like:

NAME                    CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
...
kube-dns                10.0.0.10      <none>        53/UDP,53/TCP        1h
...

If you have created the service or in the case it should be created by default but it does not appear, goto this debugging services page for more information.

Otherwise, let's go ahead.

Are dns endpoints exposed?

Varify through kubectl get endpoints command.

kubectl get ep kube-dns --namespace=kube-system

You should see something like:

NAME       ENDPOINTS                       AGE
kube-dns   10.180.3.17:53,10.180.3.17:53    1h

If not, again, goto this debugging services page and look for the endpoints section.

At last, if you reach here, the final suggestion is still goto this debugging services page and look for the "Is the kube-proxy working?" section.

Hopefully you would have got some meaningful clues now. For more Kubernetes DNS example go here.

This change is

thockin · 2016-10-13T06:15:54Z

docs/admin/dns.md

+```
+
+#### DNS known issues
+If you are using `Alpine`(lower than 3.2) as base image, dns may not work properly because `Alpine` broke dns search paths but fixed it in some later release. Please take a look at [here](https://github.com/kubernetes/kubernetes/issues/30215) for more detail.


Alpine 3.4 was when it got fixed

3.14.1 base image still not working for non-fqdn resolution

thockin · 2016-10-13T06:16:24Z

Assigning to Bowei for review

devin-donnelly · 2016-10-17T23:07:02Z

docs/admin/dns.md

@@ -182,6 +182,111 @@ Address 1: 10.0.0.1

 If you see that, DNS is working correctly.

+### Troubleshooting tips.


No periods in headers. Please change to "Troubleshooting Tips"

devin-donnelly · 2016-10-17T23:07:15Z

docs/admin/dns.md

@@ -165,7 +165,7 @@ NAME      READY     STATUS    RESTARTS   AGE
 busybox   1/1       Running   0          <some-time>
 ```

-### Validate DNS works
+### Validate DNS works.


Remove period from header. Change to "Validate that DNS is working"

devin-donnelly · 2016-10-17T23:08:13Z

docs/admin/dns.md

@@ -182,6 +182,111 @@ Address 1: 10.0.0.1

 If you see that, DNS is working correctly.

+### Troubleshooting tips.
+If above `nslookup` command does not work, here are some hints:


"If the nslookup command fails, check the following:"

devin-donnelly · 2016-10-17T23:08:25Z

docs/admin/dns.md

+If above `nslookup` command does not work, here are some hints:
+
+#### Check the local DNS configuration first
+Take a look inside the resolv.conf file. (See "Inheriting DNS from the node" and "Known issues" sections for more information)


Please provide actual links to these sections.

These two sections are currently in kubernetes/build/kube-dns/README.md, but are being removed in kubernetes/kubernetes#32931.

Don't know why they are not included in previous PR. So I moved them here as well.

devin-donnelly · 2016-10-17T23:21:54Z

docs/admin/dns.md

+Otherwise, let's go ahead.
+
+#### Are dns endpoints exposed?
+Varify through `kubectl get endpoints` command.


Varify -> "Verify"

Change to "You can verify that dns endpoints are exposed by using the kubectl get endpoints command."

devin-donnelly · 2016-10-17T23:30:05Z

docs/admin/dns.md

+See if you could find any suspicious log. If not, please continue.
+
+#### Is dns service up?
+Varify through `kubectl get service` command.


Varify -> Verify

"Verify that the DNS service is up by using the kubectl get service command.

devin-donnelly · 2016-10-17T23:30:16Z

docs/admin/dns.md

+...
+```
+
+If you have created the service or in the case it should be created by default but it does not appear, goto this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) for more information.


goto -> see

devin-donnelly · 2016-10-17T23:30:22Z

docs/admin/dns.md

+
+If you have created the service or in the case it should be created by default but it does not appear, goto this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) for more information.
+
+Otherwise, let's go ahead.


devin-donnelly · 2016-10-17T23:31:11Z

docs/admin/dns.md

+kube-dns   10.180.3.17:53,10.180.3.17:53    1h
+```
+
+If not, again, goto this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) and look for the endpoints section.


"If you do not see the endpoints, see endpoints section in the debugging services documentation."

devin-donnelly · 2016-10-17T23:33:19Z

docs/admin/dns.md

+
+At last, if you reach here, the final suggestion is still goto this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) and look for the "Is the kube-proxy working?" section.
+
+Hopefully you would have got some meaningful clues now. For more Kubernetes DNS example go [here](https://github.com/kubernetes/kubernetes/tree/master/examples/cluster-dns).


Remove first sentence.

Change second sentence to "For additional Kubernetes DNS examples, see the cluster-dns examples in the Kubernetes GitHub repository."

MrHohn · 2016-10-18T03:39:53Z

Many thanks for the detailed comments! I changed and pushed the README according to the instructions.

PS:
Both "Inheriting DNS from the node" and "Known issues" sections are currently in kubernetes/build/kube-dns/README.md, but are being removed in kubernetes/kubernetes#32931.

Don't know why they are not included in previous PR. So I moved them here as well.

devin-donnelly · 2016-10-28T03:00:05Z

@bowei, can you do a tech review on this, please?

philips · 2016-10-28T04:33:54Z

LGTM

bowei

some comment, otherwise looks good

bowei · 2016-10-28T03:17:09Z

docs/admin/dns.md

+Errors such as the following indicate a problem with the kube-dns add-on or associated Services:
+
+```
+/ # nslookup kubernetes


put kubectl exec in front of the command? may not be obvious for novices where this is run from

bowei · 2016-10-28T03:17:32Z

docs/admin/dns.md

+or
+
+```
+/ # nslookup kubernetes


kubectl exec

bowei · 2016-10-28T04:35:34Z

docs/admin/dns.md

+...
+```
+
+If you see that no pod is running or that the pod has failed/completed, the dns add-on may not be deployed by default in your current environment and you have not deployed it manually.


...current environment and you will have to deploy it manually.

bowei · 2016-10-28T04:36:17Z

docs/admin/dns.md

+
+#### Check for Errors in the DNS pod
+
+Use `kubectl logs` command for detective work.


for detective work -> to see logs for the DNS daemons

bowei · 2016-10-28T04:39:12Z

docs/admin/dns.md

+kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c healthz
+```
+
+See if you could find any suspicious log like below:


We should probably not give the example below, it is confusing -- instead you should probably say something about the log format (W, E, F letter at the beginning represent Warning, Error and Failure) and searching for entries that have these as the logging level.

bowei · 2016-10-28T04:39:52Z

docs/admin/dns.md

+F1017 17:39:44.063280   90225 server.go:56] Failed to create a kubernetes client: invalid configuration: no configuration has been provided
+```
+
+Please use [kubernetes issues](https://github.com/kubernetes/kubernetes/issues) to trace/report unintended errors.


trace/report...errors. -> report unexpected errors.

bowei · 2016-10-28T04:40:10Z

docs/admin/dns.md

+kubectl get svc --namespace=kube-system
+```
+
+You should see something like:


remove "something like"

MrHohn · 2016-10-28T16:42:29Z

Thanks for the comments! Adjusted and pushed a new commit.

bowei · 2016-10-28T18:17:07Z

LGTM

…bernetes#1449) * update 2.12 docs to use `linkerd install --crds` before `install` The Linkerd CRDs must be installed before installing the control plane. Most of the "Tasks" docs that require Linkerd to be installed just reference the "Install" task, which demonstrates the use of `install --crds`. However, some of the docs demonstrate alternative configurations or installation methods, and actually show running the `install` command. Many of those docs don't indicate that `linkerd install --crds` must be run first. This branch updates the 2.12 "Tasks" docs to show the use of `linkerd install --crds` prior to running a `linkerd install` command. I think I got all of them? I didn't add `install --crds` in a couple of places where the install command is not piped directly into `kubectl apply`. For example, in the private docker registry documentation, we show piping `linkerd install` into `grep` to get the docker image versions, and in that case, we are not actually installing Linkerd into a cluster in that command example, so the installation won't actually fail. * update helm install instructions too

Updating tags

googlebot added the cla: yes label Oct 12, 2016

MrHohn mentioned this pull request Oct 12, 2016

Add troubleshooting section to kube-dns readme kubernetes/kubernetes#31817

Closed

k8s-github-robot assigned lavalamp Oct 12, 2016

thockin reviewed Oct 13, 2016

View reviewed changes

thockin assigned bowei and unassigned lavalamp Oct 13, 2016

devin-donnelly added the Needs Tech Review label Oct 17, 2016

devin-donnelly suggested changes Oct 17, 2016

View reviewed changes

devin-donnelly added the Docs Review: Open Issues label Oct 17, 2016

MrHohn force-pushed the dns-troubleshooting branch 2 times, most recently from 0cf60a0 to 6390856 Compare October 18, 2016 03:33

devin-donnelly approved these changes Oct 28, 2016

View reviewed changes

devin-donnelly added Docs LGTM and removed Docs Review: Open Issues labels Oct 28, 2016

bowei reviewed Oct 28, 2016

View reviewed changes

Add troubleshooting section to kube-dns readme

b72518e

MrHohn force-pushed the dns-troubleshooting branch from 6390856 to b72518e Compare October 28, 2016 16:39

Merge branch 'master' into dns-troubleshooting

4b198e0

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 21, 2016

jaredbhatti merged commit fde38be into kubernetes:master Nov 21, 2016

MrHohn mentioned this pull request Dec 27, 2016

Add some troubleshooting docs to the DNS addon README.md kubernetes/kubernetes#23981

Closed

MrHohn deleted the dns-troubleshooting branch October 5, 2017 00:58

Okabe-Junya pushed a commit to Okabe-Junya/website that referenced this pull request Dec 4, 2023

Merge pull request kubernetes#1449 from cncf/CathPag-tag-update

42e9ec2

Updating tags

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add troubleshooting section to kube-dns readme #1449

Add troubleshooting section to kube-dns readme #1449

MrHohn commented Oct 12, 2016 •

edited by thockin

Loading

thockin Oct 13, 2016

Davidrjx Feb 2, 2023

thockin commented Oct 13, 2016

devin-donnelly Oct 17, 2016

devin-donnelly Oct 17, 2016

devin-donnelly Oct 17, 2016

devin-donnelly Oct 17, 2016

MrHohn Oct 18, 2016

devin-donnelly Oct 17, 2016

devin-donnelly Oct 17, 2016

devin-donnelly Oct 17, 2016

devin-donnelly Oct 17, 2016

devin-donnelly Oct 17, 2016

devin-donnelly Oct 17, 2016

MrHohn commented Oct 18, 2016 •

edited

Loading

devin-donnelly commented Oct 28, 2016

philips commented Oct 28, 2016

bowei left a comment

bowei Oct 28, 2016

bowei Oct 28, 2016

bowei Oct 28, 2016

bowei Oct 28, 2016

bowei Oct 28, 2016

bowei Oct 28, 2016

bowei Oct 28, 2016

MrHohn commented Oct 28, 2016

bowei commented Oct 28, 2016

		@@ -182,6 +182,111 @@ Address 1: 10.0.0.1

		If you see that, DNS is working correctly.

		### Troubleshooting tips.


		#### Check for Errors in the DNS pod

		Use `kubectl logs` command for detective work.

Add troubleshooting section to kube-dns readme #1449

Add troubleshooting section to kube-dns readme #1449

Conversation

MrHohn commented Oct 12, 2016 • edited by thockin Loading

Troubleshooting tips.

Check the local DNS configuration first

DNS known issues

Quick diagnosis

Is dns pod running?

Is dns pod working properly?

Is dns service up?

Are dns endpoints exposed?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Oct 13, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrHohn commented Oct 18, 2016 • edited Loading

devin-donnelly commented Oct 28, 2016

philips commented Oct 28, 2016

bowei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MrHohn commented Oct 28, 2016

bowei commented Oct 28, 2016

MrHohn commented Oct 12, 2016 •

edited by thockin

Loading

MrHohn commented Oct 18, 2016 •

edited

Loading