Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

k8s-daemonset example doesn't work on Kubernetes 1.5.1 #88

Closed
gosharplite opened this issue Feb 10, 2017 · 3 comments
Closed

k8s-daemonset example doesn't work on Kubernetes 1.5.1 #88

gosharplite opened this issue Feb 10, 2017 · 3 comments

Comments

@gosharplite
Copy link

I have a private k8s 1.5.1 cluster with three workers. I use NodePort instead of LoadBalancer in l5d service yml definition. Pods and svc are created successfully.

Just like #81, below error appears.

$ http_proxy=10.128.112.27:31396 curl -s http://hello
No hosts are available for /http/1.1/GET/hello, Dtab.base=[/srv=>/#/io.l5d.k8s/default/http;/host=>/srv;/http/*/*=>/host;/host/world=>/srv/world-v1], Dtab.local=[]. Remote Info: Not Available

10.128.112.27 is one of the worker ip, 31396 is the NodePort of l5d 4140.

$ ./kubectl describe svc l5d
Name:			l5d
Namespace:		default
Labels:			<none>
Selector:		app=l5d
Type:			NodePort
IP:			10.1.224.144
Port:			outgoing	4140/TCP
NodePort:		outgoing	31396/TCP
Endpoints:		10.128.112.26:4140,10.128.112.27:4140,10.128.112.28:4140
Port:			incoming	4141/TCP
NodePort:		incoming	31375/TCP
Endpoints:		10.128.112.26:4141,10.128.112.27:4141,10.128.112.28:4141
Port:			admin	9990/TCP
NodePort:		admin	31957/TCP
Endpoints:		10.128.112.26:9990,10.128.112.27:9990,10.128.112.28:9990
Session Affinity:	None
No events.

world-v1 pod is working since I can check it with an ubuntu pod inside k8s.

$ ./kubectl get po
NAME                                     READY     STATUS    RESTARTS   AGE
hello-851zz                              1/1       Running   0          1h
hello-dq33d                              1/1       Running   0          1h
hello-mk6m2                              1/1       Running   0          1h
l5d-8fdrr                                2/2       Running   0          1h
l5d-n8hfm                                2/2       Running   0          1h
l5d-wfrtg                                2/2       Running   0          1h
ubuntu                                   1/1       Running   0          2h
world-v1-h38bl                           1/1       Running   0          1h
world-v1-n0fql                           1/1       Running   0          1h
world-v1-x19n4                           1/1       Running   0          1h


$ ./kubectl exec -it ubuntu bash

root@ubuntu:/# curl http://world-v1:7778
world (10.2.15.174)!

hello pod is not working.

root@ubuntu:/# nslookup hello        
Server:		10.1.0.10
Address:	10.1.0.10#53

Name:	hello.default.svc.cluster.local
Address: 10.1.143.36

root@ubuntu:/# curl http://hello:7777
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

I can go into a hello pod and look around.

$ ./kubectl exec -it hello-dq33d bash

root@hello-dq33d:/usr/src/app# echo $NODE_NAME 
10.128.112.27

root@hello-dq33d:/usr/src/app# echo $POD_IP   
10.2.17.227

root@hello-dq33d:/usr/src/app# echo $http_proxy
10.128.112.27:4140

root@hello-dq33d:/usr/src/app# curl -s http://hello
No hosts are available for /http/1.1/GET/hello, Dtab.base=[/srv=>/#/io.l5d.k8s/default/http;/host=>/srv;/http/*/*=>/host;/host/world=>/srv/world-v1], Dtab.local=[]. Remote Info: Not Available

I'm stuck. Any idea how to continue?
@gosharplite
Copy link
Author

gosharplite commented Feb 10, 2017 •
edited

dtab seems to work.

dtab

Call hello directly from an ubuntu pod inside k8s does not work.

root@ubuntu:/# curl -s http://10.2.11.102:7777
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

@adleong
Copy link
Member

adleong commented Feb 10, 2017

Hi @gosharplite! Are you using CNI in your k8s cluster? If so, you'll need to set hostNetwork: true on your linkerd pods in the daemonset config as well as set hostNetwork: true on the io.l5d.k8s.daemonset transformer and io.l5d.k8s.localnode transformer in the linkerd configs.

Here's an example config that sets hostNetwork: true in those places: https://github.com/apprenda/kismatic/blob/master/ansible/roles/addon-linkerd/templates/linkerd.yml

@gosharplite
Copy link
Author

Yes, I'm using CNI. I did add hostNetwork: true in the daemonset config, but I did not set hostNetwork: true on the io.l5d.k8s.daemonset transformer and io.l5d.k8s.localnode transformer in the linkerd configs.

NODE_NAME is also needed in the daemonset config. dstPrefix: / is not needed in the linkerd configs.

After fixing daemonset and linkerd configs, expected results show up.

$ http_proxy=10.128.112.27:31396 curl -s http://hello
Hello (10.2.15.144) world (10.2.15.174)!

All yml files are here FYR.

Thanks for your help :)

@gosharplite gosharplite mentioned this issue Feb 13, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adleong @gosharplite