Service to service communication using linkerd.yml + hello-world.yml not working as expected

[jyothidat]

Hi All,

Over the past few days we have gone through several different configurations listed in following README, but we have been unable to get service to service communication between hello and world working
https://github.com/linkerd/linkerd-examples/tree/master/k8s-daemonset
We have tried hello-world.yml with one of the linkerd configuration files from the list below and in each case the hello->world communication is not working.

1. linkerd.yml
2. linkerd-ingress.yml
3. linkerd-cni.yml

Simple set of steps to reproduce this issue is given below. Please let us know a sample that works out of the box to illustrate service to service communication using linkerd as a proxy. Steps as stated on README do not work for us.

Thanks
Jyothi

----Detailed steps to reproduce the issue on minikube

1 minikube start
2 kubectl create -f linkerd.yml
3 kubectl create -f hello-world.yml
4. L5D_INGRESS_LB=$(minikube service l5d --url | head -n1)
5. Curl request direct to world works
http_proxy=$L5D_INGRESS_LB curl -s http://world
6. Curl request to hello fails
http_proxy=$L5D_INGRESS_LB curl -v -s http://hello

• Rebuilt URL to: http://hello/

GET http://hello/ HTTP/1.1
Host: hello
User-Agent: curl/7.43.0
Accept: /
Proxy-Connection: Keep-Alive

< HTTP/1.1 502 Bad Gateway
< l5d-err: No+hosts+are+available+for+%2Fsvc%2Fhello%2C+Dtab.base%3D%5B%2Fsrv%3D%3E%2F%23%2Fio.l5d.k8s%2Fdefault%2Fhttp%3B%2Fhost%3D%3E%2Fsrv%3B%2Fsvc%3D%3E%2Fhost%3B%2Fhost%2Fworld%3D%3E%2Fsrv%2Fworld-v1%5D%2C+Dtab.local%3D%5B%5D.+Remote+Info%3A+Not+Available
< Content-Type: text/plain
< Content-Length: 177
<

• Connection #0 to host left intact
  No hosts are available for /svc/hello, Dtab.base=[/srv=>/#/io.l5d.k8s/default/http;/host=>/srv;/svc=>/host;/host/world=>/srv/world-v1], Dtab.local=[]. Remote Info: Not Availablejdattatr-mac:k8s jdattatr$ http_proxy=http://:30091 curl -v -s http://world
• Rebuilt URL to: http://world/
• Trying ..
• connect to port 30091 failed: Connection refused
• Failed to connect to port 30091: Connection refused
• Closing connection 0

My minikube version is
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"fff5156092b56e6bd60fff75aad4dc9de6b6ef37", GitTreeState:"clean", BuildDate:"2017-05-09T23:22:45Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"}

[adleong]

Hi @jyothidat, is there any information in the linkerd pod's logs? And can you also provide the output of: kubectl get svc?

[esbie]

please note that as per https://discourse.linkerd.io/t/flavors-of-kubernetes, minikube does not support hello-world.yml please use hello-world-legacy.yml

[jyothidat]

@esbie Thank you - Here are some results.
1 On minikube (linkerd.yml + hello-world-legacy.yml) does work and we can curl hello and see hello-world communication.
2. On Kubernetes v 1.6 running internally (non GCE, non AWS environment) we have CNI and as such
(linkerd-cni.yml + hello-world-legacy.yml) works - implying we can curl hello and see hello-world communication.
3. On Kubernetes v 1.6 running internally (non GCE, non AWS environment) we have CNI and as such
(linkerd-cni.yml + hello-world) does not work - implying we can curl hello and see hello-world communication failing.

What are factors that we take into account to determine whether we need legacy way or non-legacy way to determine proxy hostname?

Thanks
Jyothi

[esbie]

I've written up most everything I know at https://discourse.linkerd.io/t/flavors-of-kubernetes: spec.nodeName is not available in minikube or in k8s < v1.4 and so cannot be used in the http_proxy address. In absence of this, hello-world-legacy queries the k8s api (via kubectl sidecar): https://github.com/linkerd/linkerd-examples/blob/master/docker/helloworld/hostIP.sh

I don't know a ton about CNI, so not sure if this is what's impacting your setup or if it's something else.

related: linkerd/linkerd-examples#130

[esbie]

Sorry none of these examples are working out of the box :(
Hopefully we can learn from this and improve them as a result.

[tjquinno]

@esbie It turns out that our k8s installation does not report spec.nodeName as l5d expects, and so the proxy configuration for services calling out to other services did not actually let the service talk to the proxy. We have worked around that, at least for the moment, and have seen s2s traffic working.

[esbie]

Oh! great to hear that it is working. I'm going to close this issue, please contact us in http://slack.linkerd.io/ or http://discourse.linkerd.io for additional support!