Envoy steps don't work for SDS case

[vishal-yadav]

Referring the steps from README and GKE

Till steps 3 scaling of usersvc deployment seems to work. As soon the SDS is enabled for discovery things seem to start breaking with "no healty upstream" response.

Before enabling SDS

$ kubectl get pods,svc,deploy
-----------------------------
NAME                              READY     STATUS    RESTARTS   AGE
pod/edge-envoy-59685f569f-zr4m5   1/1       Running   0          42m
pod/postgres-7c667f58f4-qtpfs     1/1       Running   0          51m
pod/usersvc-5fc9f7cb9c-gzhj5      1/1       Running   0          40m
pod/usersvc-5fc9f7cb9c-qnds4      1/1       Running   0          32m
pod/usersvc-5fc9f7cb9c-zplbn      1/1       Running   0          32m

NAME                 TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
service/edge-envoy   LoadBalancer   10.7.248.162   35.238.219.84   8000:30244/TCP   42m
service/kubernetes   ClusterIP      10.7.240.1     <none>          443/TCP          1d
service/postgres     ClusterIP      10.7.243.30    <none>          5432/TCP         51m
service/usersvc      ClusterIP      10.7.242.102   <none>          80/TCP           40m

NAME                               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/edge-envoy   1         1         1            1           42m
deployment.extensions/postgres     1         1         1            1           51m
deployment.extensions/usersvc      3         3         3            3           40m

$ curl ${ENVOY_URL}/user/alice
-----------------------------
{
  "fullname": "Alice",
  "hostname": "usersvc-5fc9f7cb9c-gzhj5",
  "ok": true,
  "resolvedname": "10.4.0.32",
  "uuid": "B30443F9F4EE446E985B57C34DB824ED"
}

$ curl ${ENVOY_URL}/user/health
-----------------------------
{
  "hostname": "usersvc-5fc9f7cb9c-gzhj5",
  "msg": "user health check OK",
  "ok": true,
  "resolvedname": "10.4.0.32"
}

$ kubectl exec usersvc-5fc9f7cb9c-gzhj5 /usr/bin/nslookup usersvc
-----------------------------
Server:         10.7.240.10
Address:        10.7.240.10#53

Non-authoritative answer:
Name:   usersvc.default.svc.cluster.local
Address: 10.7.242.102

After enabling SDS (Bring up usrsvc-sds and edge-envoy2 resources)

$ kubectl get pods,svc,deploy
NAME                               READY     STATUS    RESTARTS   AGE
pod/edge-envoy-99f8d8f4-k2xxb      1/1       Running   0          5m
pod/postgres-7c667f58f4-qtpfs      1/1       Running   0          1h
pod/usersvc-5fc9f7cb9c-gzhj5       1/1       Running   0          51m
pod/usersvc-5fc9f7cb9c-qnds4       1/1       Running   0          43m
pod/usersvc-5fc9f7cb9c-zplbn       1/1       Running   0          43m
pod/usersvc-sds-7c77bfbb96-dnkvm   1/1       Running   0          6m

NAME                  TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)          AGE
service/edge-envoy    LoadBalancer   10.7.247.222   104.155.172.67   8000:32484/TCP   5m
service/kubernetes    ClusterIP      10.7.240.1     <none>           443/TCP          1d
service/postgres      ClusterIP      10.7.243.30    <none>           5432/TCP         1h
service/usersvc       ClusterIP      10.7.242.102   <none>           80/TCP           51m
service/usersvc-sds   NodePort       10.7.243.240   <none>           5000:32128/TCP   6m

NAME                                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.extensions/edge-envoy    1         1         1            1           5m
deployment.extensions/postgres      1         1         1            1           1h
deployment.extensions/usersvc       3         3         3            3           51m
deployment.extensions/usersvc-sds   1         1         1            1           6m

$ curl -v ${ENVOY_URL}/user/health
----------------------------------
*   Trying 104.155.172.67...
* TCP_NODELAY set
* Connected to 104.155.172.67 (104.155.172.67) port 8000 (#0)
> GET /user/health HTTP/1.1
> Host: 104.155.172.67:8000
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< content-length: 19
< content-type: text/plain
< date: Tue, 12 Feb 2019 10:02:48 GMT
< server: envoy
<
* Curl_http_done: called premature == 0
* Connection #0 to host 104.155.172.67 left intact
no healthy upstream