[upgrade to 1.6.0] istiod got panic: interface conversion: proto.Message is client.QuotaSpecBinding, not client. QuotaSpecBinding #24264

pixystone · 2020-06-01T13:26:13Z

Bug description

got panic: interface conversion: proto.Message is *client.QuotaSpecBinding, not *client. QuotaSpecBinding

full logs:

panic: interface conversion: proto.Message is *client.QuotaSpecBinding, not *client.QuotaSpecBinding (types from different packages)

goroutine 945 [running]:
istio.io/istio/pilot/pkg/model.findQuotaSpecRefs(0xc0020b7354, 0x28, 0xc000f25760, 0x1, 0x1, 0x0)
        istio.io/istio/pilot/pkg/model/config.go:538 +0x2c5
istio.io/istio/pilot/pkg/model.filterQuotaSpecsByDestination(0xc0020b7354, 0x28, 0xc000f25760, 0x1, 0x1, 0xc000df4000, 0x1, 0x1, 0x0, 0xc000df4000, ...)
        istio.io/istio/pilot/pkg/model/config.go:555 +0x77
istio.io/istio/pilot/pkg/model.(*istioConfigStore).QuotaSpecByDestination(0xc000c9ed30, 0xc0020b7354, 0x28, 0x450243, 0xc0020db554, 0x288b47e)
        istio.io/istio/pilot/pkg/model/config.go:594 +0x589
istio.io/istio/pilot/pkg/networking/plugin/mixer.getQuotaSpec(0xc002068660, 0xc0020b7354, 0x28, 0xc000a4ef00, 0x0, 0x4, 0xc0020b7354)
        istio.io/istio/pilot/pkg/networking/plugin/mixer/mixer.go:710 +0x92
istio.io/istio/pilot/pkg/networking/plugin/mixer.modifyOutboundRouteConfig(0xc000de3680, 0xc002068660, 0xc0025a6660, 0x2d, 0xc0001ff180, 0x22a67c0)
        istio.io/istio/pilot/pkg/networking/plugin/mixer/mixer.go:508 +0x6a7
istio.io/istio/pilot/pkg/networking/plugin/mixer.mixerplugin.OnOutboundRouteConfiguration(0xc002068660, 0xc002104620)
        istio.io/istio/pilot/pkg/networking/plugin/mixer/mixer.go:318 +0xa2
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).buildGatewayHTTPRouteConfig(0xc00031b260, 0xc0005d7260, 0xc000de3680, 0xc000f4fbd0, 0x7, 0x1ff40c124)
        istio.io/istio/pilot/pkg/networking/core/v1alpha3/gateway.go:315 +0x123e
istio.io/istio/pilot/pkg/networking/core/v1alpha3.(*ConfigGeneratorImpl).BuildHTTPRoutes(0xc00031b260, 0xc0005d7260, 0xc000de3680, 0xc0014bacc0, 0x2, 0x2, 0x23223c0, 0x0, 0x100000000000000)
        istio.io/istio/pilot/pkg/networking/core/v1alpha3/httproute.go:68 +0x126
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).pushRoute(0xc000a84ea0, 0xc002419180, 0xc000de3680, 0xc0018ae260, 0x16, 0x2, 0xc000e9e800)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/rds.go:30 +0xb0
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).handleRds(0xc000a84ea0, 0xc002419180, 0xc000fbdc80, 0xc002419310, 0x0)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/ads.go:442 +0x25e
istio.io/istio/pilot/pkg/proxy/envoy/v2.(*DiscoveryServer).StreamAggregatedResources(0xc000a84ea0, 0x2c9ff20, 0xc00170d080, 0x0, 0x0)
        istio.io/istio/pilot/pkg/proxy/envoy/v2/ads.go:271 +0x5ca
github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2._AggregatedDiscoveryService_StreamAggregatedResources_Handler(0x2820f00, 0xc000a84ea0, 0x2c93f80, 0xc0003da000, 0x445c460, 0xc0016ca600)
        github.com/envoyproxy/go-control-plane@v0.9.5-0.20200326174812-e8bd2869ff56/envoy/service/discovery/v2/ads.pb.go:197 +0xad
google.golang.org/grpc.(*Server).processStreamingRPC(0xc000aa44e0, 0x2ca5c80, 0xc000f7ac00, 0xc0016ca600, 0xc00093fe90, 0x43fe1c0, 0x0, 0x0, 0x0)
        google.golang.org/grpc@v1.28.1/server.go:1329 +0xcdd
google.golang.org/grpc.(*Server).handleStream(0xc000aa44e0, 0x2ca5c80, 0xc000f7ac00, 0xc0016ca600, 0x0)
        google.golang.org/grpc@v1.28.1/server.go:1409 +0xc62
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001794b10, 0xc000aa44e0, 0x2ca5c80, 0xc000f7ac00, 0xc0016ca600)
        google.golang.org/grpc@v1.28.1/server.go:746 +0xa1
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.28.1/server.go:744 +0xa1

[ ] Configuration Infrastructure
[ ] Docs
[x] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

$ istioctl-1.5.4  version --remote
client version: 1.5.4
citadel version: 1.5.4
ingressgateway version: 1.6.0
ingressgateway version: 1.6.0
telemetry version: 1.6.0
pilot version: 1.5.4
istiod version:
data plane version: 1.5.1 (46 proxies), 1.5.4 (27 proxies)

How was Istio installed?

istioctl-1.6.0 install -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: dmsp-istio-control-plane
  namespace: istio-system
spec:
  revision: canary
  profile: default
  namespace: istio-system
  components:
    ingressGateways:
    - enabled: true
      name: istio-ingressgateway
    egressGateways:
    - enabled: false
      name: istio-egressgateway
    citadel:
      enabled: true
    telemetry:
      enabled: true
  addonComponents:
    grafana:
      enabled: true
    kiali:
      enabled: true
    tracing:
      enabled: true
    prometheus:
      enabled: true
  meshConfig:
    disablePolicyChecks: false
    enableTracing: true
    accessLogFile: "/dev/stdout"
    enableAutoMtls: false
    outboundTrafficPolicy:
      mode: ALLOW_ANY
    defaultConfig:
      sds:
        enabled: true
  values:  # helm values
    global:
      proxy:
        logLevel: debug
      mtls:
        enabled: false
      jwtPolicy: first-party-jwt
    telemetry:
      v1:
        enabled: true
      v2:
        enabled: false
    pilot:
      traceSampling: 1
      policy:
        enabled: true
    gateways:
      istio-ingressgateway:
        sds:
          enabled: true
EOF

Environment where bug was observed (cloud vendor, OS, etc)

macOS Catalina

NAME                                    READY   STATUS             RESTARTS   AGE
grafana-74dc798895-cdsrq                1/1     Running            0          7m59s
istio-citadel-5f4c4954-cl6kf            1/1     Running            0          11h
istio-ingressgateway-584bbf4ddf-jrq8p   1/1     Running            0          8m2s
istio-telemetry-644b5f9457-brg5q        2/2     Running            0          7m58s
istio-tracing-8584b4d7f9-bsqmv          1/1     Running            0          11h
istiod-85774f87f8-dvbkk                 1/1     Running            0          59m
istiod-canary-79599d745b-fsxk2          0/1     CrashLoopBackOff   6          8m15s
kiali-6f457f5964-9gmjh                  1/1     Running            0          37m
prometheus-5764896dfc-hdgtd             1/2     Running            0          7m59s
prometheus-6d45799d64-shrs2             2/2     Running            0          11h

The text was updated successfully, but these errors were encountered:

howardjohn · 2020-06-01T17:00:01Z

I can reproduce this. Thanks for the report

gargnupur · 2020-06-01T21:46:49Z

@pixystone : We would be fixing this issue... but also, in the mean time, for Istio 1.6, policy using mixer v1 is deprecated. (https://istio.io/docs/tasks/policy-enforcement/rate-limiting/)
You can try rate limiting using envoy's native rate limiting..
Some resources to help with the same:

pixystone · 2020-06-02T01:57:45Z

@pixystone : We would be fixing this issue... but also, in the mean time, for Istio 1.6, policy using mixer v1 is deprecated. (https://istio.io/docs/tasks/policy-enforcement/rate-limiting/)
You can try rate limiting using envoy's native rate limiting..
Some resources to help with the same:

how to configure envoyfilter to support ratelimit in istio 1.5.0？ #22068

[WIP] Add Ratelimiting test using Envoy's Ratelimit Service #23513

Thanks a lot!

~~Is this issue produced by rate limiting quota config on mixer v1?~~

I delete these configs to workaround it.

handler.config.istio.io "defaultquotahandler" deleted
instance.config.istio.io "requestcountquota" deleted
rule.config.istio.io "defaultquota" deleted
quotaspec.config.istio.io "request-count" deleted
quotaspecbinding.config.istio.io "request-count" deleted

howardjohn · 2020-07-28T22:52:54Z

This is fixed

istio-policy-bot added area/environments feature/Multi-control-plane issues related with multi-control-plane support in a cluster labels Jun 1, 2020

esnible added area/networking area/extensions and telemetry and removed area/environments feature/Multi-control-plane issues related with multi-control-plane support in a cluster labels Jun 1, 2020

howardjohn mentioned this issue Jun 1, 2020

Fix QuotaSpec crash #24276

Closed

gargnupur self-assigned this Jun 1, 2020

gargnupur mentioned this issue Jun 2, 2020

Fix QuotaSpecBinding Crash #24277

Merged

istio-testing mentioned this issue Jun 2, 2020

[release-1.6] Fix QuotaSpecBinding Crash #24359

Merged

istio-policy-bot added the lifecycle/needs-triage label Jun 5, 2020

howardjohn closed this as completed Jul 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[upgrade to 1.6.0] istiod got panic: interface conversion: proto.Message is client.QuotaSpecBinding, not client. QuotaSpecBinding #24264

[upgrade to 1.6.0] istiod got panic: interface conversion: proto.Message is client.QuotaSpecBinding, not client. QuotaSpecBinding #24264

pixystone commented Jun 1, 2020 •

edited

howardjohn commented Jun 1, 2020

gargnupur commented Jun 1, 2020

pixystone commented Jun 2, 2020 •

edited

howardjohn commented Jul 28, 2020

[upgrade to 1.6.0] istiod got panic: interface conversion: proto.Message is *client.QuotaSpecBinding, not *client. QuotaSpecBinding #24264

[upgrade to 1.6.0] istiod got panic: interface conversion: proto.Message is *client.QuotaSpecBinding, not *client. QuotaSpecBinding #24264

Comments

pixystone commented Jun 1, 2020 • edited

howardjohn commented Jun 1, 2020

gargnupur commented Jun 1, 2020

pixystone commented Jun 2, 2020 • edited

howardjohn commented Jul 28, 2020

[upgrade to 1.6.0] istiod got panic: interface conversion: proto.Message is client.QuotaSpecBinding, not client. QuotaSpecBinding #24264

[upgrade to 1.6.0] istiod got panic: interface conversion: proto.Message is client.QuotaSpecBinding, not client. QuotaSpecBinding #24264

pixystone commented Jun 1, 2020 •

edited

pixystone commented Jun 2, 2020 •

edited