Add troubleshooting section to kube-dns readme

build/kube-dns/README.md

@@ -60,7 +60,7 @@ When enabled, pods are assigned a DNS A record in the form of `pod-ip-address.my
 For example, a pod with ip `1.2.3.4` in the namespace `default` with a dns name of `cluster.local` would have an entry: `1-2-3-4.default.pod.cluster.local`.
 
 
-####A Records and hostname based on Pod's hostname and subdomain fields
+#### A Records and hostname based on Pod's hostname and subdomain fields
 Currently when a pod is created, its hostname is the Pod's `metadata.name` value.
 
 With v1.2, users can specify a Pod annotation, `pod.beta.kubernetes.io/hostname`, to specify what the Pod's hostname should be.
@@ -198,7 +198,7 @@ NAME      READY     STATUS    RESTARTS   AGE
 busybox   1/1       Running   0          <some-time>
 ```
 
-### 3 Validate DNS works
+### 3 Validate DNS works.
 Once that pod is running, you can exec nslookup in that environment:
 ```
 kubectl exec busybox -- nslookup kubernetes.default
@@ -215,6 +215,107 @@ Address 1: 10.0.0.1
 
 If you see that, DNS is working correctly.
 
+### 4 Troubleshooting tips.
+If above ```nslookup``` command does not work, here are some hints:
+
+#### Check the local DNS configuration first
+Take a look inside the resolv.conf file. (See "Inheriting DNS from the node" and "Known issues" sections for more information)
+```
+cat /etc/resolv.conf
+```
+
+You should see the search path and nameserver are set up like below: (search path may vary for different cloud providers)
+```
+search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal
+nameserver 10.0.0.10
+options ndots:5
+```
+
+#### Quick diagnosis
+If below appears, usually means something is wrong with kube-dns addon or Services
+```
+/ # nslookup kubernetes
+Server:    10.0.0.10
+Address 1: 10.0.0.10
+
+nslookup: can't resolve 'kubernetes'
+```
+
+or
+
+```
+/ # nslookup kubernetes
+Server:    10.0.0.10
+Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local
+
+nslookup: can't resolve 'kubernetes'
+```
+
+Let's continue to debug it.
+
+#### Is dns pod running?
+```kubectl get pods``` command could tell. (namespace may be "default" or others if you manually deploy it)
+```
+kubectl get pods --namespace=kube-system -a
+```
+
+You should see something like:
+```
+NAME                                                       READY     STATUS    RESTARTS   AGE
+...
+kube-dns-v19-ezo1y                                         3/3       Running   0           1h
+...
+```
+
+If no pod is running or failed/completed, then maybe because this dns addon would not be deployed by default in your current environment and you have not deployed it manually.
+
+Otherwise, please try next hint.
+
+#### Is dns pod working properly?
+Use ```kubectl logs``` command for detective work.
+```
+kubectl logs --namespace=kube-system kube-dns-v19-ezo1y -c kubedns
+kubectl logs --namespace=kube-system kube-dns-v19-ezo1y -c dnsmasq
+kubectl logs --namespace=kube-system kube-dns-v19-ezo1y -c healthz
+```
+
+See if you could find any suspicious log. If not, please continue.
+
+#### Is dns service up?
+Varify through ```kubectl get service``` command.
+```
+kubectl get svc --namespace=kube-system
+```
+
+You should see something like:
+```
+NAME                    CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+...
+kube-dns                10.0.0.10      <none>        53/UDP,53/TCP        1h
+...
+```
+
+If you have created the service or in the case it should be created by default but it does not appear, goto this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) for more information.
+
+Otherwise, let's go ahead.
+
+#### Are dns endpoints exposed?
+Varify through ```kubectl get endpoints``` command.
+```
+kubectl get ep kube-dns --namespace=kube-system
+```
+
+You should see something like:
+```
+NAME       ENDPOINTS                       AGE
+kube-dns   10.180.3.17:53,10.180.3.17:53    1h
+```
+
+If not, again, goto this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) and look for the endpoints section.
+
+At last, if you reach here, the final suggestion is still goto this [debugging services page](http://kubernetes.io/docs/user-guide/debugging-services/) and look for the "Is the kube-proxy working?" section.
+
+Hopefully you would have got some meaningful clues now. For more Kubernetes DNS example go [here](https://github.com/kubernetes/kubernetes/tree/master/examples/cluster-dns).
 
 ## How does it work?
 <del>SkyDNS depends on etcd for what to serve, but it doesn't really need all of