Preliminary ACK PR produces Envoy OOM + loads of ADS disconnects. #28192

sdake · 2020-10-22T15:15:56Z

(NOTE: This is used to report product bugs:
To report a security vulnerability, please visit https://istio.io/about/security-vulnerabilities/
To ask questions about how to use Istio, please visit https://discuss.istio.io
)

Bug description
The preliminary PR https://github.com/istio/istio/pull/27563/files leads to an OOM. The other problems I am less concerned with at this time.

Affected product area (please put an X in all that apply)

[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Affected features (please put an X in all that apply)

[ ] Multi Cluster
[ ] Virtual Machine
[ ] Multi Control Plane

Expected behavior
No OOM under heavy load.

Steps to reproduce the bug
Really hard to reproduce but is 100% reproducible. Requires 15 nodes + a unique tool. Happy to share my environment for debug via ssh public key.

Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)
master on commit ceec0d6 with above PR applied. I am preparing to rebase, so YMMV :)

How was Istio installed?

sdake@scale-cp:~$ more kn.sh
#!/bin/bash
# Install Istio and knative
./istioctl install -f istio-minimal-operator.yaml --set values.pilot.env.PILOT_XDS_SEND_TIMEOUT=35s --set values.pilot.env.PILOT_ENABLE_FLOW_CONTRO
L=true --set values.gateways.istio-egressgateway.enabled=false --set hub=quay.io/sdake --set tag=kn0338
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.17.0/serving-crds.yaml
kubectl apply --filename https://github.com/knative/serving/releases/download/v0.17.0/serving-core.yaml
kubectl apply --filename https://github.com/knative/net-istio/releases/download/v0.17.0/release.yaml

with istio-operator-minimal of:

sdake@scale-cp:~$ more istio-minimal-operator.yaml 
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      proxy:
        autoInject: disabled
      useMCP: false
  addonComponents:
    prometheus:
      enabled: false
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: false
      - name: cluster-local-gateway
        enabled: true
        label:
          istio: cluster-local-gateway
          app: cluster-local-gateway
        k8s:
          service:
            type: ClusterIP
            ports:
            - name: status-port
              port: 15021
              targetPort: 15021
            - name: http2
              port: 80
              targetPort: 8080
            - name: https
              port: 443
              targetPort: 8443

Environment where bug was observed (cloud vendor, OS, etc)
vSphere 7.0.1 15 node cluster across 3 bare metal nodes + metallb + calico + containerd + K8s 1.19.2. Each VM has 16 gb ram + 4 vcpu. K8s control plane has 32gb ram + 8 vcpu (iirc)

Additionally, please consider attaching a cluster state archive by attaching
the dump file to this issue.

cluster-local-gateway (a knative ingress) has reset.

sdake@scale-cp:~$ kubectl get pods -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
cluster-local-gateway-58b85797f-tjd7d   1/1     Running   1          26m
istiod-79f696d556-qgj2d                 1/1     Running   0          26m

As a result of SIGTERM OOMKill (137):

sdake@scale-cp:~$ kubectl describe po -n istio-system cluster-local-gateway-58b85797f-tjd7d
Name:         cluster-local-gateway-58b85797f-tjd7d
Namespace:    istio-system
Priority:     0
Node:         scale-06/192.168.7.146
Start Time:   Thu, 22 Oct 2020 14:42:39 +0000
Labels:       app=cluster-local-gateway
              chart=gateways
              heritage=Tiller
              istio=cluster-local-gateway
              pod-template-hash=58b85797f
              release=istio
              service.istio.io/canonical-name=cluster-local-gateway
              service.istio.io/canonical-revision=latest
Annotations:  cni.projectcalico.org/podIP: 172.17.25.193/32
              cni.projectcalico.org/podIPs: 172.17.25.193/32
              prometheus.io/path: /stats/prometheus
              prometheus.io/port: 15090
              prometheus.io/scrape: true
              sidecar.istio.io/inject: false
Status:       Running
IP:           172.17.25.193
IPs:
  IP:           172.17.25.193
Controlled By:  ReplicaSet/cluster-local-gateway-58b85797f
Containers:
  istio-proxy:
    Container ID:  containerd://be4ba83e2d7a75afe89610d66c428ede8b0b95f46b5b47b4f6b2d2f7f1162733
    Image:         quay.io/sdake/proxyv2:kn0338
    Image ID:      quay.io/sdake/proxyv2@sha256:807ba3e1adc65f9e5103a0bf5f72839a0d6f11e7a3a8dee07cfc534388997062
    Ports:         15021/TCP, 8080/TCP, 8443/TCP, 15090/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      proxy
      router
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
      --serviceCluster
      cluster-local-gateway
      --trust-domain=cluster.local
    State:          Running
      Started:      Thu, 22 Oct 2020 14:55:04 +0000
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Thu, 22 Oct 2020 14:42:46 +0000
      Finished:     Thu, 22 Oct 2020 14:55:03 +0000
    Ready:          True
    Restart Count:  1
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                third-party-jwt
      PILOT_CERT_PROVIDER:       istiod
      CA_ADDR:                   istiod.istio-system.svc:15012
      NODE_NAME:                  (v1:spec.nodeName)
      POD_NAME:                  cluster-local-gateway-58b85797f-tjd7d (v1:metadata.name)
      POD_NAMESPACE:             istio-system (v1:metadata.namespace)
      INSTANCE_IP:                (v1:status.podIP)
      HOST_IP:                    (v1:status.hostIP)
      SERVICE_ACCOUNT:            (v1:spec.serviceAccountName)
      CANONICAL_SERVICE:          (v1:metadata.labels['service.istio.io/canonical-name'])
      CANONICAL_REVISION:         (v1:metadata.labels['service.istio.io/canonical-revision'])
      ISTIO_META_WORKLOAD_NAME:  cluster-local-gateway
      ISTIO_META_OWNER:          kubernetes://apis/apps/v1/namespaces/istio-system/deployments/cluster-local-gateway
      ISTIO_META_MESH_ID:        cluster.local
      ISTIO_META_ROUTER_MODE:    sni-dnat
      ISTIO_META_CLUSTER_ID:     Kubernetes
    Mounts:
      /etc/istio/config from config-volume (rw)
      /etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
      /etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
      /etc/istio/pod from podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/run/ingress_gateway from gatewaysdsudspath (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-local-gateway-service-account-token-nk8wj (ro)
      /var/run/secrets/tokens from istio-token (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  gatewaysdsudspath:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio
    Optional:  true
  ingressgateway-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-certs
    Optional:    true
  ingressgateway-ca-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  istio-ingressgateway-ca-certs
    Optional:    true
  cluster-local-gateway-service-account-token-nk8wj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cluster-local-gateway-service-account-token-nk8wj
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  28m                 default-scheduler  Successfully assigned istio-system/cluster-local-gateway-58b85797f-tjd7d to scale-06
  Normal   Pulling    28m                 kubelet            Pulling image "quay.io/sdake/proxyv2:kn0338"
  Normal   Pulled     28m                 kubelet            Successfully pulled image "quay.io/sdake/proxyv2:kn0338" in 4.141367053s
  Normal   Created    15m (x2 over 28m)   kubelet            Created container istio-proxy
  Normal   Started    15m (x2 over 28m)   kubelet            Started container istio-proxy
  Normal   Pulled     15m                 kubelet            Container image "quay.io/sdake/proxyv2:kn0338" already present on machine
  Warning  Unhealthy  15m (x13 over 28m)  kubelet            Readiness probe failed: Get "http://172.17.25.193:15021/healthz/ready": dial tcp 172.17.25.193:15021: connect: connection refused

Previous logs:

sdake@scale-cp:~$ kubectl logs --previous -n istio-system cluster-local-gateway-58b85797f-tjd7d
2020-10-22T14:42:46.143467Z     info    FLAG: --concurrency="0"
2020-10-22T14:42:46.143504Z     info    FLAG: --domain="istio-system.svc.cluster.local"
2020-10-22T14:42:46.143511Z     info    FLAG: --help="false"
2020-10-22T14:42:46.143515Z     info    FLAG: --log_as_json="false"
2020-10-22T14:42:46.143518Z     info    FLAG: --log_caller=""
2020-10-22T14:42:46.143522Z     info    FLAG: --log_output_level="default:info"
2020-10-22T14:42:46.143525Z     info    FLAG: --log_rotate=""
2020-10-22T14:42:46.143543Z     info    FLAG: --log_rotate_max_age="30"
2020-10-22T14:42:46.143547Z     info    FLAG: --log_rotate_max_backups="1000"
2020-10-22T14:42:46.143551Z     info    FLAG: --log_rotate_max_size="104857600"
2020-10-22T14:42:46.143554Z     info    FLAG: --log_stacktrace_level="default:none"
2020-10-22T14:42:46.143561Z     info    FLAG: --log_target="[stdout]"
2020-10-22T14:42:46.143566Z     info    FLAG: --meshConfig="./etc/istio/config/mesh"
2020-10-22T14:42:46.143569Z     info    FLAG: --outlierLogPath=""
2020-10-22T14:42:46.143573Z     info    FLAG: --proxyComponentLogLevel="misc:error"
2020-10-22T14:42:46.143577Z     info    FLAG: --proxyLogLevel="warning"
2020-10-22T14:42:46.143581Z     info    FLAG: --serviceCluster="cluster-local-gateway"
2020-10-22T14:42:46.143585Z     info    FLAG: --stsPort="0"
2020-10-22T14:42:46.143588Z     info    FLAG: --templateFile=""
2020-10-22T14:42:46.143592Z     info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2020-10-22T14:42:46.143596Z     info    FLAG: --trust-domain="cluster.local"
2020-10-22T14:42:46.143607Z     info    Version 1.8-dev-ff1090526fc587c8546fda53bbeab2d1bb2eb682-dirty-Modified
2020-10-22T14:42:46.143780Z     info    Obtained private IP [172.17.25.193 fe80::aca5:cdff:fe0a:1c19]
2020-10-22T14:42:46.143871Z     info    Apply mesh config from file defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  proxyMetadata:
    DNS_AGENT: ""
  tracing:
    zipkin:
      address: zipkin.istio-system:9411
disableMixerHttpReports: true
enablePrometheusMerge: true
rootNamespace: istio-system
trustDomain: cluster.local
2020-10-22T14:42:46.145324Z     info    Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 0
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  DNS_AGENT: ""
serviceCluster: cluster-local-gateway
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2020-10-22T14:42:46.145382Z     info    Proxy role: &model.Proxy{RWMutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, Type:"router", IPAddresses:[]string{"172.17.25.193", "fe80::aca5:cdff:fe0a:1c19"}, ID:"cluster-local-gateway-58b85797f-tjd7d.istio-system", Locality:(*envoy_config_core_v3.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), VerifiedIdentity:(*spiffe.Identity)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), WatchedResources:map[string]*model.WatchedResource(nil)}
2020-10-22T14:42:46.145389Z     info    JWT policy is third-party-jwt
2020-10-22T14:42:46.145435Z     info    PilotSAN []string{"istiod.istio-system.svc"}
2020-10-22T14:42:46.145482Z     info    sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012 Citadel
2020-10-22T14:42:46.145527Z     info    Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2020-10-22T14:42:46.145641Z     info    citadelclient   Citadel client using custom root: istiod.istio-system.svc:15012 -----BEGIN CERTIFICATE-----
MIIC/DCCAeSgAwIBAgIQBhdBDpAo8pUA5ZpUzBTuhzANBgkqhkiG9w0BAQsFADAY
MRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTIwMTAyMjE0NDEwOFoXDTMwMTAy
MDE0NDEwOFowGDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBAMqrhiUjzGxwKpHOE/pm+0+psp5zP8OVW5Mj60sq
05rIFsU4/NSLWBJaYwHuYGO4KgWRZg81zzSLNDZ9dc+0oVdBITAbFwU76F8oHH6b
zGpKVXqN1kuyI0n+b38EZM8p9pFELofyor/b9U0klFK/OkHt4xB5qQjBjBKroBbM
awRQZkfmrMhwMnQmBohr0yZDj0BSjV64ZuqkE9kUJ4ZoblBlW89YQKthdW5+M6be
y2i+u21L2BYgZ5ZAy92nwws8lDBJKg0LJRiuWmCh+IKMKZPjhq24okLYUKs3hd4b
Yv55CYec8k0cCQLjMJrx/NU26AZlbjkm/gF01BkmZuSQqgMCAwEAAaNCMEAwDgYD
VR0PAQH/BAQDAgIEMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFNa7c24T9dcZ
ieZysW6E6MwPvbr+MA0GCSqGSIb3DQEBCwUAA4IBAQCvg0Jp6PXYgePFKgofX+jQ
QPJlzPJKAAN/xsNFdHEpGLqXc0SK6N/umpzVUxDxSVS7Kkh+cdTTa9TtAXAdX45S
7IObhuVzhrxlafPRTOrCQKhdi3mrX1CHVWypWKWFi78/mOGLa79qmS8lUbIrWF5G
A7FrADT2BB0qGIWP1oFRs77GVyyE85BvmL1RH/me8Ul/mnvNy5uDpM7EtV570ZGd
Ka0ppZSEUwJ/F+w50uWZ/XQubITN2fE9MOvENTfnnEDIQOQongLqKsnjNqi7g1PP
F3bfh5R++COzC6+8+U7UqTU7tVQwTi+ck8XaxORQm4OPrtqNEzl5iKEUbcRD5vhg
-----END CERTIFICATE-----

2020-10-22T14:42:46.179847Z     info    Starting gateway SDS
2020-10-22T14:42:46.281836Z     info    sds     SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS" 

2020-10-22T14:42:46.281961Z     info    sds     Start SDS grpc server
2020-10-22T14:42:46.282105Z     info    sds     SDS gRPC server for gateway controller starts, listening on "./var/run/ingress_gateway/sds" 

2020-10-22T14:42:46.282134Z     info    xdsproxy        Initializing with upstream address istiod.istio-system.svc:15012 and cluster Kubernetes
2020-10-22T14:42:46.282208Z     info    sds     Start SDS grpc server for ingress gateway proxy
2020-10-22T14:42:46.282398Z     info    xdsproxy        adding watcher for certificate var/run/secrets/istio/root-cert.pem
2020-10-22T14:42:46.282511Z     info    Starting proxy agent
2020-10-22T14:42:46.282523Z     info    Opening status port 15020

2020-10-22T14:42:46.282768Z     info    Received new config, creating new Envoy epoch 0
2020-10-22T14:42:46.282854Z     info    Epoch 0 starting
2020-10-22T14:42:46.357933Z     info    Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster cluster-local-gateway --service-node router~172.17.25.193~cluster-local-gateway-58b85797f-tjd7d.istio-system~istio-system.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format-prefix-with-location 0 --log-format %Y-%m-%dT%T.%fZ %l      envoy %n   %v -l warning --component-log-level misc:error]
2020-10-22T14:42:46.457335Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:42:46.516732Z     warning envoy main      there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2020-10-22T14:42:51.457655Z     error   xdsproxy        failed to connect to upstream istiod.istio-system.svc:15012: context deadline exceeded
2020-10-22T14:42:51.459308Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:42:51.905779Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:42:51.913612Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:42:52.091567Z     info    Envoy proxy is ready
2020-10-22T14:44:54.790450Z     info    sds     resource:default new connection
2020-10-22T14:44:54.790533Z     info    sds     resource:ROOTCA new connection
2020-10-22T14:44:54.790564Z     info    sds     Skipping waiting for gateway secret
2020-10-22T14:44:54.790635Z     info    sds     Skipping waiting for gateway secret
2020-10-22T14:44:55.079101Z     info    cache   Root cert has changed, start rotating root cert for SDS clients
2020-10-22T14:44:55.079149Z     info    cache   GenerateSecret default
2020-10-22T14:44:55.079672Z     info    sds     resource:default pushed key/cert pair to proxy
2020-10-22T14:44:55.392098Z     info    cache   Loaded root cert from certificate ROOTCA
2020-10-22T14:44:55.392486Z     info    sds     resource:ROOTCA pushed root cert to proxy
2020-10-22T14:48:22.688680Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:48:22.688725Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:48:22.689191Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:48:23.094956Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:48:23.104743Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:48:41.761076Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:48:41.761127Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:48:41.761674Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:48:42.152915Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:48:42.163245Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:48:56.048642Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.route.v3.RouteConfiguration: context deadline exceeded
2020-10-22T14:48:56.048716Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:48:56.049018Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:48:56.392275Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:48:58.400098Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:49:09.724484Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:49:09.724566Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:49:09.724971Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:49:09.767794Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:49:09.787322Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:49:25.972490Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:49:25.972558Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:49:25.972853Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:49:26.186643Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:49:26.196808Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:49:37.747950Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:49:37.747993Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:49:37.748976Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:49:38.129311Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:49:40.138794Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:49:54.995170Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:49:54.995258Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:49:54.995605Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:49:55.192377Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:49:55.211692Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:05.201292Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:50:05.201352Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:05.201754Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:50:05.684172Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:50:05.695697Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:17.790820Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:50:17.790863Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:17.791189Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:50:17.827970Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:50:17.845136Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:25.360387Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:50:25.360419Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:25.360678Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:50:25.718617Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:50:25.728791Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:37.787763Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:50:37.787821Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:37.788294Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:50:38.027647Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:50:38.039440Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:49.195058Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:50:49.195135Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:50:49.195417Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:50:49.348168Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:50:49.368401Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:01.264285Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:51:01.264333Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:01.264593Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:51:01.638904Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:51:01.660126Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:14.638897Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:51:14.638969Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:14.639466Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:51:15.149880Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:51:15.157742Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:23.800809Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:51:23.800896Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:23.801919Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:51:23.892678Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:51:23.912933Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:34.420212Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:51:34.420255Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:34.420519Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:51:34.836123Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:51:34.847440Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:42.979311Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:51:42.981397Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:42.981729Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:51:43.408714Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:51:43.419257Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:53.574346Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:51:53.574413Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:51:53.574788Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:51:54.014776Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:51:54.026470Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:02.733215Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:52:02.733262Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:02.733833Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:52:03.164556Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:52:03.172579Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:11.699602Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:52:11.699652Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:11.699844Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:52:12.025169Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:52:12.035785Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:30.086488Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:52:30.086543Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:30.087199Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:52:30.452467Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:52:30.482865Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:38.776768Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:52:38.776809Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:38.777371Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:52:39.264049Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:52:39.273580Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:51.546045Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:52:51.546129Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:52:51.546513Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:52:51.792755Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:52:51.804920Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:00.815792Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:53:00.815855Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:00.816203Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:53:01.100835Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:53:01.112608Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:09.535183Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:53:09.535235Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:09.535694Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:53:09.690904Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:53:09.702479Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:19.330527Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:53:19.330568Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:19.337246Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:53:19.828523Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:53:19.840359Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:30.686932Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:53:30.687017Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:30.687354Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:53:31.002530Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:53:31.020320Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:38.982022Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:53:38.982160Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:38.982448Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:53:39.196400Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:53:39.215763Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:51.878341Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:53:51.878387Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:53:51.880910Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:53:52.148077Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:53:52.157735Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:04.393711Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:54:04.393762Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:04.396062Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:54:04.574562Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:54:04.590710Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:12.518602Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.route.v3.RouteConfiguration: context deadline exceeded
2020-10-22T14:54:12.518644Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:12.519183Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:54:12.658664Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:54:12.672525Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:23.406806Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:54:23.406884Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:23.407392Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:54:23.787433Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:54:23.799078Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:34.329966Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:54:34.330014Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:34.330344Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:54:34.632398Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:54:34.651357Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:46.140645Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:54:46.140696Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:46.141140Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:54:46.251512Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:54:46.268568Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:56.201509Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:54:56.201570Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:54:56.202119Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:54:56.654017Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:54:56.665611Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012

Current Envoy logs:

sdake@scale-cp:~$ kubectl logs -n istio-system cluster-local-gateway-58b85797f-tjd7d
2020-10-22T14:55:04.908222Z     info    FLAG: --concurrency="0"
2020-10-22T14:55:04.908261Z     info    FLAG: --domain="istio-system.svc.cluster.local"
2020-10-22T14:55:04.908268Z     info    FLAG: --help="false"
2020-10-22T14:55:04.908273Z     info    FLAG: --log_as_json="false"
2020-10-22T14:55:04.908282Z     info    FLAG: --log_caller=""
2020-10-22T14:55:04.908287Z     info    FLAG: --log_output_level="default:info"
2020-10-22T14:55:04.908291Z     info    FLAG: --log_rotate=""
2020-10-22T14:55:04.908304Z     info    FLAG: --log_rotate_max_age="30"
2020-10-22T14:55:04.908309Z     info    FLAG: --log_rotate_max_backups="1000"
2020-10-22T14:55:04.908313Z     info    FLAG: --log_rotate_max_size="104857600"
2020-10-22T14:55:04.908321Z     info    FLAG: --log_stacktrace_level="default:none"
2020-10-22T14:55:04.908329Z     info    FLAG: --log_target="[stdout]"
2020-10-22T14:55:04.908334Z     info    FLAG: --meshConfig="./etc/istio/config/mesh"
2020-10-22T14:55:04.908342Z     info    FLAG: --outlierLogPath=""
2020-10-22T14:55:04.908347Z     info    FLAG: --proxyComponentLogLevel="misc:error"
2020-10-22T14:55:04.908351Z     info    FLAG: --proxyLogLevel="warning"
2020-10-22T14:55:04.908359Z     info    FLAG: --serviceCluster="cluster-local-gateway"
2020-10-22T14:55:04.908364Z     info    FLAG: --stsPort="0"
2020-10-22T14:55:04.908368Z     info    FLAG: --templateFile=""
2020-10-22T14:55:04.908376Z     info    FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2020-10-22T14:55:04.908388Z     info    FLAG: --trust-domain="cluster.local"
2020-10-22T14:55:04.908409Z     info    Version 1.8-dev-ff1090526fc587c8546fda53bbeab2d1bb2eb682-dirty-Modified
2020-10-22T14:55:04.908773Z     info    Obtained private IP [172.17.25.193 fe80::aca5:cdff:fe0a:1c19]
2020-10-22T14:55:04.908895Z     info    Apply mesh config from file defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  proxyMetadata:
    DNS_AGENT: ""
  tracing:
    zipkin:
      address: zipkin.istio-system:9411
disableMixerHttpReports: true
enablePrometheusMerge: true
rootNamespace: istio-system
trustDomain: cluster.local
2020-10-22T14:55:04.911330Z     info    Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 0
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  DNS_AGENT: ""
serviceCluster: cluster-local-gateway
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2020-10-22T14:55:04.911383Z     info    Proxy role: &model.Proxy{RWMutex:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, Type:"router", IPAddresses:[]string{"172.17.25.193", "fe80::aca5:cdff:fe0a:1c19"}, ID:"cluster-local-gateway-58b85797f-tjd7d.istio-system", Locality:(*envoy_config_core_v3.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), VerifiedIdentity:(*spiffe.Identity)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), WatchedResources:map[string]*model.WatchedResource(nil)}
2020-10-22T14:55:04.911390Z     info    JWT policy is third-party-jwt
2020-10-22T14:55:04.911437Z     info    PilotSAN []string{"istiod.istio-system.svc"}
2020-10-22T14:55:04.911488Z     info    sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012 Citadel
2020-10-22T14:55:04.911536Z     info    Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2020-10-22T14:55:04.911647Z     info    citadelclient   Citadel client using custom root: istiod.istio-system.svc:15012 -----BEGIN CERTIFICATE-----
MIIC/DCCAeSgAwIBAgIQBhdBDpAo8pUA5ZpUzBTuhzANBgkqhkiG9w0BAQsFADAY
MRYwFAYDVQQKEw1jbHVzdGVyLmxvY2FsMB4XDTIwMTAyMjE0NDEwOFoXDTMwMTAy
MDE0NDEwOFowGDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcN
AQEBBQADggEPADCCAQoCggEBAMqrhiUjzGxwKpHOE/pm+0+psp5zP8OVW5Mj60sq
05rIFsU4/NSLWBJaYwHuYGO4KgWRZg81zzSLNDZ9dc+0oVdBITAbFwU76F8oHH6b
zGpKVXqN1kuyI0n+b38EZM8p9pFELofyor/b9U0klFK/OkHt4xB5qQjBjBKroBbM
awRQZkfmrMhwMnQmBohr0yZDj0BSjV64ZuqkE9kUJ4ZoblBlW89YQKthdW5+M6be
y2i+u21L2BYgZ5ZAy92nwws8lDBJKg0LJRiuWmCh+IKMKZPjhq24okLYUKs3hd4b
Yv55CYec8k0cCQLjMJrx/NU26AZlbjkm/gF01BkmZuSQqgMCAwEAAaNCMEAwDgYD
VR0PAQH/BAQDAgIEMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFNa7c24T9dcZ
ieZysW6E6MwPvbr+MA0GCSqGSIb3DQEBCwUAA4IBAQCvg0Jp6PXYgePFKgofX+jQ
QPJlzPJKAAN/xsNFdHEpGLqXc0SK6N/umpzVUxDxSVS7Kkh+cdTTa9TtAXAdX45S
7IObhuVzhrxlafPRTOrCQKhdi3mrX1CHVWypWKWFi78/mOGLa79qmS8lUbIrWF5G
A7FrADT2BB0qGIWP1oFRs77GVyyE85BvmL1RH/me8Ul/mnvNy5uDpM7EtV570ZGd
Ka0ppZSEUwJ/F+w50uWZ/XQubITN2fE9MOvENTfnnEDIQOQongLqKsnjNqi7g1PP
F3bfh5R++COzC6+8+U7UqTU7tVQwTi+ck8XaxORQm4OPrtqNEzl5iKEUbcRD5vhg
-----END CERTIFICATE-----

2020-10-22T14:55:04.953697Z     info    Starting gateway SDS
2020-10-22T14:55:05.056125Z     info    sds     SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS" 

2020-10-22T14:55:05.056266Z     info    sds     SDS gRPC server for gateway controller starts, listening on "./var/run/ingress_gateway/sds" 

2020-10-22T14:55:05.056297Z     info    xdsproxy        Initializing with upstream address istiod.istio-system.svc:15012 and cluster Kubernetes
2020-10-22T14:55:05.056629Z     info    xdsproxy        adding watcher for certificate var/run/secrets/istio/root-cert.pem
2020-10-22T14:55:05.056860Z     info    Starting proxy agent
2020-10-22T14:55:05.056988Z     info    sds     Start SDS grpc server for ingress gateway proxy
2020-10-22T14:55:05.057008Z     info    Received new config, creating new Envoy epoch 0
2020-10-22T14:55:05.057099Z     info    Opening status port 15020

2020-10-22T14:55:05.057208Z     info    Epoch 0 starting
2020-10-22T14:55:05.057072Z     info    sds     Start SDS grpc server
2020-10-22T14:55:05.141048Z     info    Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster cluster-local-gateway --service-node router~172.17.25.193~cluster-local-gateway-58b85797f-tjd7d.istio-system~istio-system.svc.cluster.local --local-address-ip-version v4 --bootstrap-version 3 --log-format-prefix-with-location 0 --log-format %Y-%m-%dT%T.%fZ %l      envoy %n   %v -l warning --component-log-level misc:error]
2020-10-22T14:55:05.281074Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:55:05.281287Z     warning envoy main      there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2020-10-22T14:55:05.296553Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:55:10.743225Z     info    sds     resource:ROOTCA new connection
2020-10-22T14:55:10.743379Z     info    sds     Skipping waiting for gateway secret
2020-10-22T14:55:10.743730Z     info    sds     resource:default new connection
2020-10-22T14:55:10.744013Z     info    sds     Skipping waiting for gateway secret
2020-10-22T14:55:11.179723Z     info    cache   Root cert has changed, start rotating root cert for SDS clients
2020-10-22T14:55:11.179779Z     info    cache   GenerateSecret default
2020-10-22T14:55:11.180486Z     info    sds     resource:default pushed key/cert pair to proxy
2020-10-22T14:55:11.343561Z     info    cache   Loaded root cert from certificate ROOTCA
2020-10-22T14:55:11.344811Z     info    sds     resource:ROOTCA pushed root cert to proxy
2020-10-22T14:55:18.062249Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.cluster.v3.Cluster: context deadline exceeded
2020-10-22T14:55:18.062312Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:55:18.063101Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:55:18.422161Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:55:18.441796Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:55:26.196113Z     info    Envoy proxy is ready
2020-10-22T14:55:31.686068Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:55:31.686142Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:55:31.686419Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:55:32.102001Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:55:32.114191Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:55:42.100129Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:55:42.100187Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:55:42.101210Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:55:42.314876Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:55:42.324792Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:55:59.090566Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:55:59.090626Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:55:59.090951Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:55:59.569274Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:55:59.587309Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:56:09.743774Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:56:09.743841Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:56:09.744441Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:56:09.777563Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:56:09.797743Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:56:23.533016Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:56:23.533055Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:56:23.539443Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:56:24.047890Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:56:24.060641Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:56:36.564731Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:56:36.564771Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:56:36.565245Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:56:37.114099Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:56:37.138570Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:56:49.675906Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:56:49.675993Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:56:49.676309Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:56:49.909289Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:56:49.928735Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:57:05.115986Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:57:05.116033Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:57:05.116289Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:57:05.149083Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:57:05.183573Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:57:18.062229Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:57:18.062484Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:57:18.063045Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:57:18.552274Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:57:18.560503Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:57:32.061750Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:57:32.061809Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:57:32.062124Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:57:32.174253Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:57:32.187761Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:58:16.549506Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:58:16.549567Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:58:16.550853Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:58:16.956015Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:58:16.969451Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:58:25.810933Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:58:25.810974Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:58:25.811365Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:58:25.911753Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:58:25.926626Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:58:42.110076Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment: context deadline exceeded
2020-10-22T14:58:42.110144Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:58:42.110926Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:58:42.382139Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:58:42.393452Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T14:58:58.899758Z     error   xdsproxy        upstream send error for type url type.googleapis.com/envoy.config.listener.v3.Listener: context deadline exceeded
2020-10-22T14:58:58.899834Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T14:58:58.900387Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 2, context deadline exceeded
2020-10-22T14:58:59.126914Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T14:58:59.137204Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T15:01:45.375560Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T15:01:45.376033Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 0, 
2020-10-22T15:01:45.395648Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T15:01:45.409123Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T15:03:46.967585Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T15:03:46.968012Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 0, 
2020-10-22T15:03:47.112279Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T15:03:47.124468Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T15:05:48.935847Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T15:05:48.936185Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 0, 
2020-10-22T15:05:49.165858Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T15:05:49.185671Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T15:07:50.926762Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T15:07:50.927120Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 0, 
2020-10-22T15:07:51.052859Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T15:07:51.063502Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T15:09:52.532953Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T15:09:52.533484Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 0, 
2020-10-22T15:09:52.726665Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T15:09:52.735542Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012
2020-10-22T15:11:54.144517Z     info    xdsproxy        disconnected from XDS server: istiod.istio-system.svc:15012
2020-10-22T15:11:54.144999Z     warning envoy config    StreamAggregatedResources gRPC config stream closed: 0, 
2020-10-22T15:11:54.655830Z     info    xdsproxy        Envoy ADS stream established
2020-10-22T15:11:54.664880Z     info    xdsproxy        connecting to upstream XDS server: istiod.istio-system.svc:15012

Pilot logs attached.
log.tar.gz

The text was updated successfully, but these errors were encountered:

sdake · 2020-10-22T21:40:03Z

@howardjohn logs for that envoy OOM. Smells like a memory leak to me, but I don't know enough about Envoy to judge.

the disconnecting ADS is probably related to the flow control not rate limiting prpoerly.

Cheers,
-steve

howardjohn · 2020-10-22T22:31:55Z

@sdake one possible thing is to try PROXY_XDS_VIA_AGENT=false env var on istio

this is the new xds proxy we have added. One thing I noticed is it also has a 5s timeout hardcoded (this is for calls from envoy to istiod), so it may be undoing the increased timeout you did

sdake · 2020-10-23T00:36:28Z

Turning this off causes flow control to have no visual effect, but also stops crashing the ADS connection and the OOMs stop. I attempted to increase the 5s hardcode to 35s, and same result.

It appears the knative benchmark has found an Istio master bug. I wonder how much trouble it would be to run this in the long-running scale/perf tests?

Its sort of like wackamole on this flow control - hard to get to a stable baseline 😫

Cheers,
-steve

lut777 · 2020-12-22T04:14:53Z

It looks like I'm having the same problem in istio 1.8.1.
The log shows that xds connection is disconnected every once in a while. Then after several disconnections, OOM suddenly appears. 1024Mi memory doesn't mean anything here.
I'm wondering if these multiple disconnections of xds are the cause of OOM? If I follow @sdake 's configuration and change the hardtime to 30 seconds, can this OOM be avoided?

howardjohn · 2020-12-22T04:16:29Z

this is about an off by default experimental feature, probably not your issue. I would get a heap profile and open a new issue (https://github.com/istio/istio/wiki/Analyzing-Istio-Performance).

lut777 · 2020-12-22T04:21:16Z

Great. glad to see pprof's showing up.

sdake · 2021-01-03T10:09:41Z

John, This issue exhibited in the past without the flow control flag enabled. It had the appearance of being fixed in 1.8 - although I didn't heavily verify. Cheers, -steve

…

On Mon, Dec 21, 2020 at 9:21 PM lut777 ***@***.***> wrote: Great. glad to see pprof's showing up. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#28192 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFYRCKBKXKJLMOPSSLQEGTSWANEXANCNFSM4S3K6OKQ> .

Minh-Ng · 2021-03-03T18:01:09Z

Seeing the same symptoms on 1.8.3, disabling the xDS proxy via agent solved the issue.

istio-policy-bot · 2021-04-19T05:33:07Z

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2021-01-03. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

This was referenced Oct 22, 2020

Wait to send more pushes until receive has ACKed #27563

Closed

Istio is picking up new virtualservice slowly #25685

Closed

sdake added the area/networking/envoy label Oct 22, 2020

istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Apr 4, 2021

istio-policy-bot closed this as completed Apr 19, 2021

istio-policy-bot added the lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. label Apr 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preliminary ACK PR produces Envoy OOM + loads of ADS disconnects. #28192

Preliminary ACK PR produces Envoy OOM + loads of ADS disconnects. #28192

sdake commented Oct 22, 2020 •

edited

sdake commented Oct 22, 2020

howardjohn commented Oct 22, 2020

sdake commented Oct 23, 2020

lut777 commented Dec 22, 2020 •

edited

howardjohn commented Dec 22, 2020 •

edited

lut777 commented Dec 22, 2020

sdake commented Jan 3, 2021 via email

Minh-Ng commented Mar 3, 2021

istio-policy-bot commented Apr 19, 2021

Preliminary ACK PR produces Envoy OOM + loads of ADS disconnects. #28192

Preliminary ACK PR produces Envoy OOM + loads of ADS disconnects. #28192

Comments

sdake commented Oct 22, 2020 • edited

sdake commented Oct 22, 2020

howardjohn commented Oct 22, 2020

sdake commented Oct 23, 2020

lut777 commented Dec 22, 2020 • edited

howardjohn commented Dec 22, 2020 • edited

lut777 commented Dec 22, 2020

sdake commented Jan 3, 2021 via email

Minh-Ng commented Mar 3, 2021

istio-policy-bot commented Apr 19, 2021

sdake commented Oct 22, 2020 •

edited

lut777 commented Dec 22, 2020 •

edited

howardjohn commented Dec 22, 2020 •

edited