Internal gRPC connections not working

[stevemartinaws]

Is this a BUG or FEATURE REQUEST?:

Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?:

Yes

Bug:
Y

What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details

istioctl version 0.5.1
kubectl version 0.5.1

Is Istio Auth enabled or not ?
No

What happened:
Node gRPC app cannot communicate with go gRPC services internally - this works if using the external NodePort
Worked in 0.0.4

This is the error from the node client process

 { Error: 14 UNAVAILABLE: Connect Failed
    at new createStatusError (/usr/src/app/node_modules/grpc/src/client.js:64:15)
    at /usr/src/app/node_modules/grpc/src/client.js:583:15
  code: 14,
  metadata: Metadata { _internal_repr: {} },
  details: 'Connect Failed' }

I have removed the sidecars from the apps and everything is communicated as expected

What you expected to happen:
We should be able to communicate between services using internal DNS without having to route outside the cluster

How to reproduce it:
run a node gRPCclient calling golang gRPC server, both using sidecar

[ldemailly]

are you using the correct service name and port for your server ? any relevant logs on source (and destination) istio-proxy containers?

[stevemartinaws]

service names and ports are correct, although these are not fully qualified with the namespace. Everything is in default and we have never had to do it.

I have also attempted to use the ClusterIP from the service to connect without success.

I have done some more testing and as soon as I introduce the sidecar again my webapplication probes fail and nothing can communicate (the probes depend on backend connectivity)

Backend gRPC service Logs

2018-02-20T18:59:16.562209Z	info	Proxy role: model.Node{Type:"sidecar", IPAddress:"10.2.6.128", ID:"authsvc-86df94ccd6-slvgh.default", Domain:"default.svc.cluster.local"}
2018-02-20T18:59:16.562225Z	info	Attempting to lookup address: istio-mixer.istio-system
2018-02-20T18:59:16.564000Z	info	Addr resolved to: 10.3.60.122:9125
2018-02-20T18:59:16.564025Z	info	Finished lookup of address: istio-mixer.istio-system
{
  "listeners": [],
  "lds": {
    "cluster": "lds",
    "refresh_delay_ms": 1000
  },
  "admin": {
    "access_log_path": "/dev/stdout",
    "address": "tcp://127.0.0.1:15000"
  },
  "cluster_manager": {
    "clusters": [
      {
        "name": "rds",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://istio-pilot.istio-system:15003"
          }
        ]
      },
      {
        "name": "lds",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://istio-pilot.istio-system:15003"
          }
        ]
      },
      {
        "name": "zipkin",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://zipkin.istio-system:9411"
          }
        ]
      }
    ],
    "sds": {
      "cluster": {
        "name": "sds",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://istio-pilot.istio-system:15003"
          }
        ]
      },
      "refresh_delay_ms": 1000
    },
    "cds": {
      "cluster": {
        "name": "cds",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://istio-pilot.istio-system:15003"
          }
        ]
      },
      "refresh_delay_ms": 1000
    }
  },
  "statsd_udp_ip_address": "10.3.60.122:9125",
  "tracing": {
    "http": {
      "driver": {
        "type": "zipkin",
        "config": {
          "collector_cluster": "zipkin",
          "collector_endpoint": "/api/v1/spans"
        }
      }
    }
  }
2018-02-20T18:59:16.564548Z	info	Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 1.000s
discoveryAddress: istio-pilot.istio-system:15003
discoveryRefreshDelay: 1.000s
drainDuration: 2.000s
parentShutdownDuration: 3.000s
proxyAdminPort: 15000
serviceCluster: authsvc
statsdUdpAddress: 10.3.60.122:9125
zipkinAddress: zipkin.istio-system:9411
2018-02-20T18:59:16.564567Z	info	Monitored certs: []envoy.CertSource{}
2018-02-20T18:59:16.564690Z	info	Starting proxy agent
2018-02-20T18:59:16.564761Z	info	Received new config, resetting budget
2018-02-20T18:59:16.564770Z	info	Reconciling configuration (budget 10)
2018-02-20T18:59:16.564787Z	info	Epoch 0 starting
2018-02-20T18:59:16.564816Z	info	writing configuration to /etc/istio/proxy/envoy-rev0.json
2018-02-20T18:59:16.566212Z	info	Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 2 --parent-shutdown-time-s 3 --service-cluster authsvc --service-node sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local --max-obj-name-len 189 -l off]
2018-02-20T18:59:16.566513Z	info	Availability zone not set, proxy will default to not using zone aware routing. To manually override use the --availabilityZone flag.
2018-02-20T18:59:46.570604Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: EOF
2018-02-20T19:00:16.574020Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: EOF
2018-02-20T19:00:46.577467Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: EOF
2018-02-20T19:01:16.581107Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: read tcp 10.2.6.128:54528->10.3.104.126:15003: read: connection reset by peer
2018-02-20T19:01:46.584006Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: EOF
2018-02-20T19:02:16.586973Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: EOF
2018-02-20T19:02:46.589903Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: read tcp 10.2.6.128:56350->10.3.104.126:15003: read: connection reset by peer
2018-02-20T19:03:16.592955Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: read tcp 10.2.6.128:56944->10.3.104.126:15003: read: connection reset by peer
2018-02-20T19:03:46.595333Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: read tcp 10.2.6.128:57518->10.3.104.126:15003: read: connection reset by peer
2018-02-20T19:04:16.599457Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/authsvc/sidecar~10.2.6.128~authsvc-86df94ccd6-slvgh.default~default.svc.cluster.local: read tcp 10.2.6.128:581

Backend gRPC service Yaml

kind: Deployment
apiVersion: apps/v1beta2
metadata:
  annotations:
    deployment.kubernetes.io/revision: '5'
    selfLink: /apis/apps/v1beta2/namespaces/default/deployments/authsvc
  resourceVersion: '28214578'
  name: authsvc
  uid: 320b04c5-10ce-11e8-8b5a-00155d00c95b
  creationTimestamp: '2018-02-13T14:57:22Z'
  generation: 5
  namespace: default
  labels:
    app: authsvc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: authsvc
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: authsvc
      annotations:
        sidecar.istio.io/status: >-
          {"version":"2bd86b8d0885ea10d7c6d64e1a090f6c6be3a110e29bdb04c9a0ed3e12105730","initContainers":["istio-init","enable-core-dump"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs"]}
    spec:
      restartPolicy: Always
      initContainers:
        - name: istio-init
          image: 'docker.io/istio/proxy_init:0.5.1'
          args:
            - '-p'
            - '15001'
            - '-u'
            - '1337'
            - '-i'
            - '10.2.0.0/16,10.3.0.0/16'
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
        - resources: {}
          terminationMessagePath: /dev/termination-log
          name: enable-core-dump
          command:
            - /bin/sh
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          terminationMessagePolicy: File
          image: alpine
          args:
            - '-c'
            - >-
              sysctl -w kernel.core_pattern=/etc/istio/proxy/core.%e.%p.%t &&
              ulimit -c unlimited
      imagePullSecrets:
        - name: registrypullsecret
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      securityContext: {}
      containers:
        - resources: {}
          terminationMessagePath: /dev/termination-log
          name: authsvc
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8086
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 1
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 3
          env:
            - name: DEPLOY_ENV
              value: development
            - name: SITE_SQL_CONNECTION_STRING
              value: >-
                server=XXXXXX;Initial
                Catalog=XXXXXXX;encrypt=disable;user
                id=XXXXXX;password=XXXXXXX;port=1433
            - name: PORT
              value: '8080'
          ports:
            - containerPort: 8080
              protocol: TCP
          imagePullPolicy: Always
          terminationMessagePolicy: File
          image: registry.gitlab.com/awsltd/authsvc
        - resources: {}
          terminationMessagePath: /dev/termination-log
          name: istio-proxy
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: INSTANCE_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
          securityContext:
            privileged: false
            runAsUser: 1337
            readOnlyRootFilesystem: true
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: istio-envoy
              mountPath: /etc/istio/proxy
            - name: istio-certs
              readOnly: true
              mountPath: /etc/certs/
          terminationMessagePolicy: File
          image: 'docker.io/istio/proxy:0.5.1'
          args:
            - proxy
            - sidecar
            - '--configPath'
            - /etc/istio/proxy
            - '--binaryPath'
            - /usr/local/bin/envoy
            - '--serviceCluster'
            - authsvc
            - '--drainDuration'
            - 2s
            - '--parentShutdownDuration'
            - 3s
            - '--discoveryAddress'
            - 'istio-pilot.istio-system:15003'
            - '--discoveryRefreshDelay'
            - 1s
            - '--zipkinAddress'
            - 'zipkin.istio-system:9411'
            - '--connectTimeout'
            - 1s
            - '--statsdUdpAddress'
            - 'istio-mixer.istio-system:9125'
            - '--proxyAdminPort'
            - '15000'
            - '--controlPlaneAuthPolicy'
            - NONE
      volumes:
        - name: istio-envoy
          emptyDir:
            medium: Memory
        - name: istio-certs
          secret:
            secretName: istio.default
            defaultMode: 420
            optional: true
      dnsPolicy: ClusterFirst
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 2
  progressDeadlineSeconds: 600
status:
  observedGeneration: 5
  replicas: 1
  updatedReplicas: 1
  readyReplicas: 1
  availableReplicas: 1
  conditions:
    - type: Progressing
      status: 'True'
      lastUpdateTime: '2018-02-20T18:59:17Z'
      lastTransitionTime: '2018-02-13T14:57:22Z'
      reason: NewReplicaSetAvailable
      message: ReplicaSet "authsvc-86df94ccd6" has successfully progressed.
    - type: Available
      status: 'True'
      lastUpdateTime: '2018-02-20T19:19:09Z'
      lastTransitionTime: '2018-02-20T19:19:09Z'
      reason: MinimumReplicasAvailable
      message: Deployment has minimum availability.

Web Application Logs

2018-02-20T16:08:06.626356Z	info	Version root@2e4a18076b04-docker.io/istio-0.5.1-30acfe6528107ea333543309095659b93364b30d-Clean
2018-02-20T16:08:06.626399Z	info	Proxy role: model.Node{Type:"sidecar", IPAddress:"10.2.2.114", ID:"mywebapplication-5cf646f7cb-gz8m6.default", Domain:"default.svc.cluster.local"}
2018-02-20T16:08:06.626408Z	info	Attempting to lookup address: istio-mixer.istio-system
2018-02-20T16:08:06.628794Z	info	Addr resolved to: 10.3.60.122:9125
2018-02-20T16:08:06.628814Z	info	Finished lookup of address: istio-mixer.istio-system
2018-02-20T16:08:06.629171Z	info	Effective config: binaryPath: /usr/local/bin/envoy
configPath: /etc/istio/proxy
connectTimeout: 1.000s
discoveryAddress: istio-pilot.istio-system:15003
discoveryRefreshDelay: 1.000s
drainDuration: 2.000s
parentShutdownDuration: 3.000s
proxyAdminPort: 15000
serviceCluster: mywebapplication
statsdUdpAddress: 10.3.60.122:9125
zipkinAddress: zipkin.istio-system:9411
2018-02-20T16:08:06.629183Z	info	Monitored certs: []envoy.CertSource{}
2018-02-20T16:08:06.629425Z	info	Starting proxy agent
2018-02-20T16:08:06.629481Z	info	Received new config, resetting budget
2018-02-20T16:08:06.629490Z	info	Reconciling configuration (budget 10)
2018-02-20T16:08:06.629519Z	info	Availability zone not set, proxy will default to not using zone aware routing. To manually override use the --availabilityZone flag.
2018-02-20T16:08:06.629546Z	info	Epoch 0 starting
2018-02-20T16:08:06.629577Z	info	writing configuration to /etc/istio/proxy/envoy-rev0.json
{
  "listeners": [],
  "lds": {
    "cluster": "lds",
    "refresh_delay_ms": 1000
  },
  "admin": {
    "access_log_path": "/dev/stdout",
    "address": "tcp://127.0.0.1:15000"
  },
  "cluster_manager": {
    "clusters": [
      {
        "name": "rds",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://istio-pilot.istio-system:15003"
2018-02-20T16:08:06.630167Z	info	Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 2 --parent-shutdown-time-s 3 --service-cluster mywebapplication --service-node sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local --max-obj-name-len 189 -l off]
          }
        ]
      },
      {
        "name": "lds",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://istio-pilot.istio-system:15003"
          }
        ]
      },
      {
        "name": "zipkin",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://zipkin.istio-system:9411"
          }
        ]
      }
    ],
    "sds": {
      "cluster": {
        "name": "sds",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://istio-pilot.istio-system:15003"
          }
        ]
      },
      "refresh_delay_ms": 1000
    },
    "cds": {
      "cluster": {
        "name": "cds",
        "connect_timeout_ms": 1000,
        "type": "strict_dns",
        "lb_type": "round_robin",
        "hosts": [
          {
            "url": "tcp://istio-pilot.istio-system:15003"
          }
        ]
      },
      "refresh_delay_ms": 1000
    }
  },
  "statsd_udp_ip_address": "10.3.60.122:9125",
  "tracing": {
    "http": {
      "driver": {
        "type": "zipkin",
        "config": {
          "collector_cluster": "zipkin",
          "collector_endpoint": "/api/v1/spans"
        }
      }
    }
  }
2018-02-20T16:08:36.631871Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: EOF
2018-02-20T16:09:06.638530Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: EOF
2018-02-20T16:09:36.641228Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: read tcp 10.2.2.114:59638->10.3.104.126:15003: read: connection reset by peer
2018-02-20T16:10:06.644097Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: read tcp 10.2.2.114:60046->10.3.104.126:15003: read: connection reset by peer
2018-02-20T16:10:36.646786Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: EOF
2018-02-20T16:11:06.649457Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: EOF
2018-02-20T16:11:36.651822Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: EOF
2018-02-20T16:12:06.654927Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: read tcp 10.2.2.114:33466->10.3.104.126:15003: read: connection reset by peer
2018-02-20T16:12:36.657435Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: EOF
2018-02-20T16:13:06.659911Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: EOF
2018-02-20T16:13:36.662440Z	info	Unable to retrieve availability zone from pilot: Get http://istio-pilot.istio-system:15003/v1/az/mywebapplication/sidecar~10.2.2.114~mywebapplication-5cf646f7cb-gz8m6.default~default.svc.cluster.local: read tcp 10.2.2.114:34730->10.3.104.126:15003: read: connection reset by peer
}[libprotobuf INFO external/mixerclient_git/src/check_cache.cc:155] Add a new Referenced for check cache: Absence-keys: Exact-keys: context.protocol, destination.ip, destination.service, destination.uid, source.ip, source.uid, 
[2018-02-20T18:42:00.660Z] "GET / HTTP/1.1" 200 - 0 7230 71 68 "10.10.0.1, 10.2.3.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "e8a43765-f97a-98e0-be1d-9574959705ad" "mywebaaplication.com" "127.0.0.1:8080"
[libprotobuf INFO external/mixerclient_git/src/check_cache.cc:155] Add a new Referenced for check cache: Absence-keys: Exact-keys: context.protocol, destination.ip, destination.service, destination.uid, source.ip, source.uid, 
[libprotobuf INFO external/mixerclient_git/src/check_cache.cc:155] Add a new Referenced for check cache: Absence-keys: Exact-keys: context.protocol, destination.ip, destination.service, destination.uid, source.ip, source.uid, 
[libprotobuf INFO external/mixerclient_git/src/check_cache.cc:155] Add a new Referenced for check cache: Absence-keys: Exact-keys: context.protocol, destination.ip, destination.service, destination.uid, source.ip, source.uid, 
[2018-02-20T19:00:30.209Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.6.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "efd93769-a7a9-91b5-bbe4-a4551a37044b" "mywebaaplication.com" "-"
[2018-02-20T19:04:10.901Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.3.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "d8be0a01-e6f4-99d7-acbd-1134d007fe67" "mywebaaplication.com" "-"
[2018-02-20T19:04:12.424Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "f878c1e2-a65c-9bbf-84b9-dd7a42629fad" "mywebaaplication.com" "-"
[2018-02-20T19:04:12.590Z] "GET /favicon.ico HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.6.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "55633c6d-dfde-9e4d-95ad-15f09504ed7f" "mywebaaplication.com" "-"
[2018-02-20T19:04:12.628Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.3.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "0ee945f2-19cf-9016-8231-7327c1b36bfe" "mywebaaplication.com" "-"
[2018-02-20T19:04:12.784Z] "GET /favicon.ico HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "029a1e5f-eb8b-9412-918d-6f0228469708" "mywebaaplication.com" "-"
[2018-02-20T19:04:12.813Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.6.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "37bc1636-3607-97e5-b983-689079564457" "mywebaaplication.com" "-"
[2018-02-20T19:04:12.985Z] "GET /favicon.ico HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.3.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "981e4f51-419f-9aea-94ad-7b8f1eb65377" "mywebaaplication.com" "-"
[2018-02-20T19:04:12.997Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "94b67afd-6866-9eaa-988b-dea4d20d4fb6" "mywebaaplication.com" "-"
[2018-02-20T19:04:13.184Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.6.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "5fe538c7-0527-9a3b-8e7f-f4fd9ad11bd1" "mywebaaplication.com" "-"
[2018-02-20T19:04:13.382Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.3.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "7a9daea0-fa35-91b0-8c22-8ba32d8ce805" "mywebaaplication.com" "-"
[2018-02-20T19:04:13.573Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "5af8eed9-1e30-95e8-95f9-ca056e6d67

Web Application Config

kind: Deployment
apiVersion: apps/v1beta2
metadata:
  annotations:
    deployment.kubernetes.io/revision: '5'
    selfLink: /apis/apps/v1beta2/namespaces/default/deployments/mywebapplication
  resourceVersion: '28210553'
  name: mywebapplication
  uid: ebf6fabe-11a5-11e8-9b53-00155d00c95b
  creationTimestamp: '2018-02-14T16:41:36Z'
  generation: 5
  namespace: default
  labels:
    type: devbranch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mywebapplication
  template:
    metadata:
      creationTimestamp: null
      labels:
        version: 08244960efb347632b529e6aadfea225b3dbffde
        app: mywebapplication
      annotations:
        sidecar.istio.io/status: >-
          {"version":"2bd86b8d0885ea10d7c6d64e1a090f6c6be3a110e29bdb04c9a0ed3e12105730","initContainers":["istio-init","enable-core-dump"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-certs"]}
    spec:
      restartPolicy: Always
      initContainers:
        - name: istio-init
          image: 'docker.io/istio/proxy_init:0.5.1'
          args:
            - '-p'
            - '15001'
            - '-u'
            - '1337'
            - '-i'
            - '10.2.0.0/16,10.3.0.0/16'
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
        - resources: {}
          terminationMessagePath: /dev/termination-log
          name: enable-core-dump
          command:
            - /bin/sh
          securityContext:
            privileged: true
          imagePullPolicy: IfNotPresent
          terminationMessagePolicy: File
          image: alpine
          args:
            - '-c'
            - >-
              sysctl -w kernel.core_pattern=/etc/istio/proxy/core.%e.%p.%t &&
              ulimit -c unlimited
      imagePullSecrets:
        - name: registrypullsecret
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 30
      securityContext: {}
      containers:
        - resources: {}
          readinessProbe:
            exec:
              command:
                - curl
                - '-f'
                - 'http://localhost:8087/probes/readyz'
            initialDelaySeconds: 10
            timeoutSeconds: 1
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          name: mywebapplication
          livenessProbe:
            exec:
              command:
                - curl
                - '-f'
                - 'http://localhost:8087/probes/healthz'
            initialDelaySeconds: 90
            timeoutSeconds: 1
            periodSeconds: 30
            successThreshold: 1
            failureThreshold: 3
          env:
            - name: DEPLOY_DATE
              value: 'Tue Feb 20 16:07:32 UTC 2018'
            - name: AUTHSVC
              value: 'authsvc:8080'
            - name: COOKIE_DOMAIN
              value: .mywebapplication.com
            - name: NODE_ENV
              value: local
            - name: CMSSVC
              value: 'cmssvc:8075'
            - name: API_ENV
              value: mywebapplication
          ports:
            - containerPort: 8080
              protocol: TCP
          imagePullPolicy: Always
          terminationMessagePolicy: File
          image: 'myregistry.com/mywebapplication:latest'
        - resources: {}
          terminationMessagePath: /dev/termination-log
          name: istio-proxy
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: INSTANCE_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
          securityContext:
            privileged: false
            runAsUser: 1337
            readOnlyRootFilesystem: true
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: istio-envoy
              mountPath: /etc/istio/proxy
            - name: istio-certs
              readOnly: true
              mountPath: /etc/certs/
          terminationMessagePolicy: File
          image: 'docker.io/istio/proxy:0.5.1'
          args:
            - proxy
            - sidecar
            - '--configPath'
            - /etc/istio/proxy
            - '--binaryPath'
            - /usr/local/bin/envoy
            - '--serviceCluster'
            - mywebapplication
            - '--drainDuration'
            - 2s
            - '--parentShutdownDuration'
            - 3s
            - '--discoveryAddress'
            - 'istio-pilot.istio-system:15003'
            - '--discoveryRefreshDelay'
            - 1s
            - '--zipkinAddress'
            - 'zipkin.istio-system:9411'
            - '--connectTimeout'
            - 1s
            - '--statsdUdpAddress'
            - 'istio-mixer.istio-system:9125'
            - '--proxyAdminPort'
            - '15000'
            - '--controlPlaneAuthPolicy'
            - NONE
      volumes:
        - name: istio-envoy
          emptyDir:
            medium: Memory
        - name: istio-certs
          secret:
            secretName: istio.default
            defaultMode: 420
            optional: true
      dnsPolicy: ClusterFirst
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
status:
  observedGeneration: 5
  replicas: 1
  updatedReplicas: 1
  unavailableReplicas: 1
  conditions:
    - type: Progressing
      status: 'True'
      lastUpdateTime: '2018-02-20T16:21:43Z'
      lastTransitionTime: '2018-02-20T16:21:43Z'
      reason: NewReplicaSetAvailable
      message: >-
        ReplicaSet "mywebapplication-5cf646f7cb" has successfully
        progressed.
    - type: Available
      status: 'False'
      lastUpdateTime: '2018-02-20T19:00:22Z'
      lastTransitionTime: '2018-02-20T19:00:22Z'
      reason: MinimumReplicasUnavailable
      message: Deployment does not have minimum availability.

Ingress Logs

[libprotobuf ERROR external/mixerclient_git/src/check_cache.cc:147] Referenced attributes: Absence-keys: destination.ip, source.uid, target.ip, Exact-keys: context.protocol, destination.service, destination.uid, source.ip, 
[libprotobuf ERROR external/mixerclient_git/src/check_cache.cc:145] Response referenced mismatchs with request
[libprotobuf ERROR external/mixerclient_git/src/check_cache.cc:146] Request attributes: attributes {
  key: "context.protocol"
  value {
    string_value: "http"
  }
}
attributes {
  key: "destination.ip"
  value {
  }
}
attributes {
  key: "destination.service"
  value {
    string_value: "istio-ingress.istio-system.svc.cluster.local"
  }
}
attributes {
  key: "destination.uid"
  value {
    string_value: "kubernetes://istio-ingress-65798d998f-5wsq9.istio-system"
  }
}
attributes {
  key: "request.headers"
  value {
    string_map_value {
      entries {
        key: ":authority"
        value: "mywebapplication.com"
      }
      entries {
        key: ":method"
        value: "GET"
      }
      entries {
        key: ":path"
        value: "/"
      }
      entries {
        key: "accept"
        value: "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
      }
      entries {
        key: "accept-encoding"
        value: "gzip, deflate"
      }
      entries {
        key: "accept-language"
        value: "en-GB,en-US;q=0.9,en;q=0.8"
      }
      entries {
        key: "cache-control"
        value: "max-age=0"
      }
      entries {
        key: "cookie"
        value: "CameViaEntryPage=CountryID=158&GMTOffset=0&Time=20/02/2018+18:42:02; _ga=GA1.2.505739743.1519152122; _gid=GA1.2.1003795510.1519152122"
      }
      entries {
        key: "dnt"
        value: "1"
      }
      entries {
        key: "upgrade-insecure-requests"
        value: "1"
      }
      entries {
        key: "user-agent"
        value: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36"
      }
      entries {
        key: "x-b3-sampled"
        value: "1"
      }
      entries {
        key: "x-b3-spanid"
        value: "4cd9f4c5b3b11591"
      }
      entries {
        key: "x-b3-traceid"
        value: "4cd9f4c5b3b11591"
      }
      entries {
        key: "x-envoy-decorator-operation"
        value: "default-route"
      }
      entries {
        key: "x-envoy-external-address"
        value: "10.2.3.0"
      }
      entries {
        key: "x-forwarded-for"
        value: "10.10.0.1, 10.2.3.0"
      }
      entries {
        key: "x-forwarded-host"
        value: "*.adultlite.live"
      }
      entries {
        key: "x-forwarded-proto"
        value: "http"
      }
      entries {
        key: "x-ot-span-context"
        value: "4cd9f4c5b3b11591;4cd9f4c5b3b11591;0000000000000000"
      }
      entries {
        key: "x-real-ip"
        value: "10.10.0.1"
      }
      entries {
        key: "x-request-id"
        value: "7a9daea0-fa35-91b0-8c22-8ba32d8ce805"
      }
    }
  }
}
attributes {
  key: "request.host"
  value {
    string_value: "mywebapplication.com"
  }
}
attributes {
  key: "request.method"
  value {
    string_value: "GET"
  }
}
attributes {
  key: "request.path"
  value {
    string_value: "/"
  }
}
attributes {
  key: "request.scheme"
  value {
    string_value: "http"
  }
}
attributes {
  key: "request.time"
  value {
    timestamp_value {
      seconds: 1519153453
      nanos: 381353599
    }
  }
}
attributes {
  key: "request.useragent"
  value {
    string_value: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36"
  }
}
attributes {
  key: "source.ip"
  value {
    bytes_value: "\n\002\003\000"
  }
}
attributes {
  key: "source.port"
  value {
    int64_value: 43154
  }
}
[libprotobuf ERROR external/mixerclient_git/src/check_cache.cc:147] Referenced attributes: Absence-keys: destination.ip, source.uid, target.ip, Exact-keys: context.protocol, destination.service, destination.uid, source.ip, 
[libprotobuf ERROR external/mixerclient_git/src/check_cache.cc:145] Response referenced mismatchs with request
[libprotobuf ERROR external/mixerclient_git/src/check_cache.cc:146] Request attributes: attributes {
  key: "context.protocol"
  value {
    string_value: "http"
  }
}
attributes {
  key: "destination.ip"
  value {
  }
}
attributes {
  key: "destination.service"
  value {
    string_value: "istio-ingress.istio-system.svc.cluster.local"
  }
}
attributes {
  key: "destination.uid"
  value {
    string_value: "kubernetes://istio-ingress-65798d998f-5wsq9.istio-system"
  }
}
attributes {
  key: "request.headers"
  value {
    string_map_value {
      entries {
        key: ":authority"
        value: "mywebapplication.com"
      }
      entries {
        key: ":method"
        value: "GET"
      }
      entries {
        key: ":path"
        value: "/"
      }
      entries {
        key: "accept"
        value: "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"
      }
      entries {
        key: "accept-encoding"
        value: "gzip, deflate"
      }
      entries {
        key: "accept-language"
        value: "en-GB,en-US;q=0.9,en;q=0.8"
      }
      entries {
        key: "cache-control"
        value: "max-age=0"
      }
      entries {
        key: "cookie"
        value: "CameViaEntryPage=CountryID=158&GMTOffset=0&Time=20/02/2018+18:42:02; _ga=GA1.2.505739743.1519152122; _gid=GA1.2.1003795510.1519152122"
      }
      entries {
        key: "dnt"
        value: "1"
      }
      entries {
        key: "upgrade-insecure-requests"
        value: "1"
      }
      entries {
        key: "user-agent"
        value: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36"
      }
      entries {
        key: "x-b3-sampled"
        value: "1"
      }
      entries {
        key: "x-b3-spanid"
        value: "2446f1cae8106ede"
      }
      entries {
        key: "x-b3-traceid"
        value: "2446f1cae8106ede"
      }
      entries {
        key: "x-envoy-decorator-operation"
        value: "default-route"
      }
      entries {
        key: "x-envoy-external-address"
        value: "10.2.2.0"
      }
      entries {
        key: "x-forwarded-for"
        value: "10.10.0.1, 10.2.2.0"
      }
      entries {
        key: "x-forwarded-host"
        value: "*.adultlite.live"
      }
      entries {
        key: "x-forwarded-proto"
        value: "http"
      }
      entries {
        key: "x-ot-span-context"
        value: "2446f1cae8106ede;2446f1cae8106ede;0000000000000000"
      }
      entries {
        key: "x-real-ip"
        value: "10.10.0.1"
      }
      entries {
        key: "x-request-id"
        value: "5af8eed9-1e30-95e8-95f9-ca056e6d6745"
      }
    }
  }
}
attributes {
  key: "request.host"
  value {
    string_value: "mywebapplication.com"
  }
}
attributes {
  key: "request.method"
  value {
    string_value: "GET"
  }
}
attributes {
  key: "request.path"
  value {
    string_value: "/"
  }
}
attributes {
  key: "request.scheme"
  value {
    string_value: "http"
  }
}
attributes {
  key: "request.time"
  value {
    timestamp_value {
      seconds: 1519153453
      nanos: 571405294
    }
  }
}
attributes {
  key: "request.useragent"
  value {
    string_value: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36"
  }
}
attributes {
  key: "source.ip"
  value {
    bytes_value: "\n\002\002\000"
  }
}
attributes {
  key: "source.port"
  value {
    int64_value: 52566
  }
}
[libprotobuf ERROR external/mixerclient_git/src/check_cache.cc:147] Referenced attributes: Absence-keys: destination.ip, source.uid, target.ip, Exact-keys: context.protocol, destination.service, destination.uid, source.ip, 
[2018-02-20T19:04:12.423Z] "GET / HTTP/1.1" 404 - 0 0 2 0 "10.10.0.1, 10.2.2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "f878c1e2-a65c-9bbf-84b9-dd7a42629fad" "mywebapplication.com" "10.2.2.114:8080"
[2018-02-20T19:04:12.589Z] "GET /favicon.ico HTTP/1.1" 404 - 0 0 2 0 "10.10.0.1, 10.2.6.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "55633c6d-dfde-9e4d-95ad-15f09504ed7f" "mywebapplication.com" "10.2.2.114:8080"
[2018-02-20T19:04:12.627Z] "GET / HTTP/1.1" 404 - 0 0 2 0 "10.10.0.1, 10.2.3.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "0ee945f2-19cf-9016-8231-7327c1b36bfe" "mywebapplication.com" "10.2.2.114:8080"
[2018-02-20T19:04:12.783Z] "GET /favicon.ico HTTP/1.1" 404 - 0 0 2 0 "10.10.0.1, 10.2.2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "029a1e5f-eb8b-9412-918d-6f0228469708" "mywebapplication.com" "10.2.2.114:8080"
[2018-02-20T19:04:12.812Z] "GET / HTTP/1.1" 404 - 0 0 2 1 "10.10.0.1, 10.2.6.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "37bc1636-3607-97e5-b983-689079564457" "mywebapplication.com" "10.2.2.114:8080"
[2018-02-20T19:04:12.984Z] "GET /favicon.ico HTTP/1.1" 404 - 0 0 2 0 "10.10.0.1, 10.2.3.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "981e4f51-419f-9aea-94ad-7b8f1eb65377" "mywebapplication.com" "10.2.2.114:8080"
[2018-02-20T19:04:12.997Z] "GET / HTTP/1.1" 404 - 0 0 1 0 "10.10.0.1, 10.2.2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "94b67afd-6866-9eaa-988b-dea4d20d4fb6" "mywebapplication.com" "10.2.2.114:8080"
[2018-02-20T19:04:13.183Z] "GET / HTTP/1.1" 404 - 0 0 2 0 "10.10.0.1, 10.2.6.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "5fe538c7-0527-9a3b-8e7f-f4fd9ad11bd1" "mywebapplication.com" "10.2.2.114:8080"
[2018-02-20T19:04:13.381Z] "GET / HTTP/1.1" 404 - 0 0 2 0 "10.10.0.1, 10.2.3.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "7a9daea0-fa35-91b0-8c22-8ba32d8ce805" "mywebapplication.com" "10.2.2.114:8080"
[2018-02-20T19:04:13.571Z] "GET / HTTP/1.1" 404 - 0 0 3 0 "10.10.0.1, 10.2.2.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "5af8eed9-1e30-95e8-95f9-ca056e6d6745"

[ldemailly]

it seems your app is asking for mywebaaplication.com which isn't known to pilot/envoy ?

[2018-02-20T19:00:30.209Z] "GET / HTTP/1.1" 404 NR 0 0 0 - "10.10.0.1, 10.2.6.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" "efd93769-a7a9-91b5-bbe4-a4551a37044b" "mywebaaplication.com" "-"

[stevemartinaws]

traffic comes in via an ingress rule based on host mywebaaplication.com. As the app cannot communicate with backend services the health and readiness checks do not pass and the pod never becomes ready. If I remove these checks I then get logs from the gRPC client stating that the connection has failed.

[ldemailly]

can you capture the log from doing just 1 grpc call from inside the mesh grpc client to mesh grpc server and show only the logs for that ? (not ingress etc...)

if you have grpc health check you could use fortio grpcping -health svc:port to test too

[ldemailly]

going to close this as we have grpc tests passing

if you have issues with 0.8 please open a new issue