Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modifying gateway servers causes envoy PROTOCOL ERROR, excessive ADS pushes #13739

Closed
howardjohn opened this issue Apr 30, 2019 · 1 comment

Comments

1 participant
@howardjohn
Copy link
Member

commented Apr 30, 2019

On 1.1.4

These both repeats a few times a second, for roughly a minute (I think its related to the drain time):
Pilot Logs:

2019-04-30T22:44:41.758990Z     info    ads     CDS: PUSH 2019-04-30T22:43:57Z/45 for router~10.60.56.28~istio-ingressgateway-655dd667cc-f45jc.istio-system~istio-system.svc.cluster.local-1335 "10.60.56.28:42506", Clusters: 70, Services 15
2019-04-30T22:44:41.760257Z     info    ads     LDS: PUSH for node:istio-ingressgateway-655dd667cc-f45jc.istio-system addr:"10.60.56.28:42506" listeners:1 1042
2019-04-30T22:44:41.762063Z     error   buildGatewayRoutes: could not find server for routeName http.80, have map[]
2019-04-30T22:44:41.762176Z     warn    ads     RDS: Failed to generate routes for route http.80 for node &{Kubernetes router [10.60.56.28] istio-ingressgateway-655dd667cc-f45jc.istio-system region:"us-central1" zone:"us-central1-a"  istio-system.svc.cluster.local istio-system  map[istio:sidecar CONFIG_NAMESPACE:istio-system ISTIO_PROXY_VERSION:1.1.3 ISTIO_PROXY_SHA:istio-proxy:73fa9b1f29f91029cc2485a685994a0d1dbcde21 ISTIO_VERSION:1.1.4 ROUTER_MODE:sni-dnat ISTIO_META_INSTANCE_IPS:10.60.56.28,10.60.56.28,fe80::a8b5:87ff:fe3d:1d8c POD_NAME:istio-ingressgateway-655dd667cc-f45jc] <nil> [0xc000830a80 0xc000830b00 0xc000830b80 0xc000830c00 0xc000830c80 0xc000830e00 0xc000830f00 0xc000831000 0xc000831180] [app=istio-ingressgateway,chart=gateways,heritage=Tiller,istio=ingressgateway,pod-template-hash=655dd667cc,release=istio]}: buildGatewayRoutes: could not find server for routeName http.80, have map[]
2019-04-30T22:44:41.762330Z     error   ads     ADS: "10.60.56.28:42506" router~10.60.56.28~istio-ingressgateway-655dd667cc-f45jc.istio-system~istio-system.svc.cluster.local-1335 terminated with stream closed

Proxy logs:

[2019-04-30 22:38:00.276][19][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 2, RDS: Failed to generate route http.80 for node &{Kubernetes router [10.60.56.28] istio-ingressgateway-655dd667cc-f45jc.istio-system region:"us-central1" zone:"us-central1-a"  istio-system.svc.cluster.local istio-system  map[POD_NAME:istio-ingressgateway-655dd667cc-f45jc istio:sidecar CONFIG_NAMESPACE:istio-system ISTIO_PROXY_VERSION:1.1.3 ISTIO_PROXY_SHA:istio-proxy:73fa9b1f29f91029cc2485a685994a0d1dbcde21 ISTIO_VERSION:1.1.4 ROUTER_MODE:sni-dnat ISTIO_META_INSTANCE_IPS:10.60.56.28,10.60.56.28,fe80::a8b5:87ff:fe3d:1d8c] <nil> [0xc000856500 0xc000856600 0xc000856700 0xc000856900 0xc000856a00 0xc000856b00 0xc000856b80 0xc000856e00 0xc000856f00] [app=istio-ingressgateway,chart=gateways,heritage=Tiller,istio=ingressgateway,pod-template-hash=655dd667cc,release=istio]}: buildGatewayRoutes: could not find server for routeName http.80, have map[]

Pilot finally ends with

2019-04-30T22:44:42.135994Z     warn    ads     ADS:RDS: ACK PROTOCOL ERROR 10.60.56.28:42506 router~10.60.56.28~istio-ingressgateway-655dd667cc-f45jc.istio-system~istio-system.svc.cluster.local-1336 (&{Kubernetes router [10.60.56.28] istio-ingressgateway-655dd667cc-f45jc.istio-system region:"us-central1" zone:"us-central1-a"  istio-system.svc.cluster.local istio-system  map[CONFIG_NAMESPACE:istio-system ISTIO_PROXY_VERSION:1.1.3 ISTIO_PROXY_SHA:istio-proxy:73fa9b1f29f91029cc2485a685994a0d1dbcde21 ISTIO_VERSION:1.1.4 ROUTER_MODE:sni-dnat ISTIO_META_INSTANCE_IPS:10.60.56.28,10.60.56.28,fe80::a8b5:87ff:fe3d:1d8c POD_NAME:istio-ingressgateway-655dd667cc-f45jc istio:sidecar] <nil> [0xc000549c80 0xc000549d00 0xc000549d80 0xc000549f00 0xc000842000 0xc000842080 0xc000842100 0xc000842280 0xc000842380] [app=istio-ingressgateway,chart=gateways,heritage=Tiller,istio=ingressgateway,pod-template-hash=655dd667cc,release=istio]}) version_info:"2019-04-30T22:43:51Z/44" node:<id:"router~10.60.56.28~istio-ingressgateway-655dd667cc-f45jc.istio-system~istio-system.svc.cluster.local" cluster:"istio-ingressgateway" metadata:<fields:<key:"CONFIG_NAMESPACE" value:<string_value:"istio-system" > > fields:<key:"ISTIO_META_INSTANCE_IPS" value:<string_value:"10.60.56.28,10.60.56.28,fe80::a8b5:87ff:fe3d:1d8c" > > fields:<key:"ISTIO_PROXY_SHA" value:<string_value:"istio-proxy:73fa9b1f29f91029cc2485a685994a0d1dbcde21" > > fields:<key:"ISTIO_PROXY_VERSION" value:<string_value:"1.1.3" > > fields:<key:"ISTIO_VERSION" value:<string_value:"1.1.4" > > fields:<key:"POD_NAME" value:<string_value:"istio-ingressgateway-655dd667cc-f45jc" > > fields:<key:"ROUTER_MODE" value:<string_value:"sni-dnat" > > fields:<key:"istio" value:<string_value:"sidecar" > > > locality:<> build_version:"73fa9b1f29f91029cc2485a685994a0d1dbcde21/1.11.0-dev/Clean/RELEASE/BoringSSL" > type_url:"type.googleapis.com/envoy.api.v2.RouteConfiguration" response_nonce:"dfb7a8a4-82cd-4055-b552-664a6e505e83"

Steps to reproduce:
Apply this config:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.example.com
  gateways:
  - mygateway
  http:
  - match:
    - port: 80
    route:
    - destination:
        host: my-httpbin
        port:
          number: 80
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mygateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - httpbin.example.com

Then apply this config:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: other
spec:
  hosts:
  - other.example.com
  gateways:
  - mygateway
  tcp:
  - match:
    - port: 5555
    route:
    - destination:
        host: my-other
        port:
          number: 5555
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mygateway
spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
  servers:
  - port:
      number: 5555
      name: tcp
      protocol: TCP
    hosts:
    - other.example.com

Aside from major log spam, it is also triggering a new push ~5 times a second

@howardjohn howardjohn changed the title Modifying gateway servers causes Modifying gateway servers causes envoy PROTOCOL ERROR, excessive ADS pushes Apr 30, 2019

@howardjohn howardjohn self-assigned this May 8, 2019

@howardjohn

This comment has been minimized.

Copy link
Member Author

commented May 11, 2019

Fixed in 1.1.6

@howardjohn howardjohn closed this May 11, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.