SDS/RDS sends two DiscoveryRequest with same config resource name #5744

quanjielin · 2019-01-29T01:16:32Z

envoy use config file
config_dump.txt
in the config, there are only two unique sdsconfig (default and ROOTCA); so server side should receive two stream requests(one for default and one for ROOTCA) when envoy start (envoy may send more requests later depends on server side resp)

      "tls_context": {
       "common_tls_context": {
        "alpn_protocols": [
         "istio",
         "h2"
        ],
        "tls_certificate_sds_secret_configs": [
         {
          "name": "default",
          "sds_config": {
           "api_config_source": {
            "api_type": "GRPC",
            "grpc_services": [
             {
              "google_grpc": {
               "target_uri": "unix:/var/run/sds/uds_path",
               "channel_credentials": {
                "local_credentials": {}
               },
               "call_credentials": [
                {
                 "from_plugin": {
                  "name": "envoy.grpc_credentials.file_based_metadata",
                  "config": {
                   "secret_data": {
                    "filename": "/var/run/secrets/kubernetes.io/serviceaccount/token"
                   },
                   "header_key": "istio_sds_credentail_header-bin"
                  }
                 }
                }
               ],
               "stat_prefix": "sdsstat",
               "credentials_factory_name": "envoy.grpc_credentials.file_based_metadata"
              }
             }
            ]
           }
          }
         }
        ],
        "combined_validation_context": {
         "default_validation_context": {
          "verify_subject_alt_name": [
           "spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account"
          ]
         },
         "validation_context_sds_secret_config": {
          "name": "ROOTCA",
          "sds_config": {
           "api_config_source": {
            "api_type": "GRPC",
            "grpc_services": [
             {
              "google_grpc": {
               "target_uri": "unix:/var/run/sds/uds_path",
               "channel_credentials": {
                "local_credentials": {}
               },
               "call_credentials": [
                {
                 "from_plugin": {
                  "name": "envoy.grpc_credentials.file_based_metadata",
                  "config": {
                   "secret_data": {
                    "filename": "/var/run/secrets/kubernetes.io/serviceaccount/token"
                   },
                   "header_key": "istio_sds_credentail_header-bin"
                  }
                 }
                }
               ],
               "stat_prefix": "sdsstat",
               "credentials_factory_name": "envoy.grpc_credentials.file_based_metadata"
              }
             }
            ]
           }
          }
         }
        }
       },

however, from both envoy log
envoy1.log
and server side log, envoy send two requests(for same sdsconfig resource name), almost at the same time when new pod starts.

The question is why envoy sends two same requests for same sdsconfig resource ?

envoy logs

[2019-01-28 23:00:25.890][23][trace][upstream] [external/envoy/source/common/config/grpc_mux_impl.cc:63] draining discovery requests 1
[2019-01-28 23:00:25.890][23][trace][upstream] [external/envoy/source/common/config/grpc_mux_impl.cc:114] Sending DiscoveryRequest for type.googleapis.com/envoy.api.v2.auth.Secret: node {
  id: "router~10.8.2.83~istio-ingressgateway-6bd597dc46-sfc4t.istio-system~istio-system.svc.cluster.local"
  cluster: "istio-ingressgateway"
  metadata {
    fields {
      key: "CONFIG_NAMESPACE"
      value {
        string_value: "istio-system"
      }
    }
    fields {
      key: "ISTIO_META_INSTANCE_IPS"
      value {
        string_value: "10.8.2.83,10.8.2.83,fe80::3017:e8ff:feab:f9a8"
      }
    }
    fields {
      key: "ISTIO_PROXY_SHA"
      value {
        string_value: "istio-proxy:7738fa3d506d3a1a7be002c1f06377e6516bc245"
      }
    }
    fields {
      key: "ISTIO_PROXY_VERSION"
      value {
        string_value: "1.1.0"
      }
    }
    fields {
      key: "ISTIO_VERSION"
      value {
        string_value: "1.0-dev"
      }
    }
    fields {
      key: "POD_NAME"
      value {
        string_value: "istio-ingressgateway-6bd597dc46-sfc4t"
      }
    }
    fields {
      key: "ROUTER_MODE"
      value {
        string_value: "sni-dnat"
      }
    }
    fields {
      key: "istio"
      value {
        string_value: "sidecar"
      }
    }
  }
  locality {
  }
  build_version: "7738fa3d506d3a1a7be002c1f06377e6516bc245/1.10.0-dev/Clean/RELEASE/BoringSSL"
}
resource_names: "default"
type_url: "type.googleapis.com/envoy.api.v2.auth.Secret"

[2019-01-28 23:00:25.890][23][trace][grpc] [external/envoy/source/common/grpc/google_async_client_impl.cc:197] Queued message to write (609 bytes)
[2019-01-28 23:00:25.890][23][debug][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:60] gRPC config for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment accepted with 1 resources: [cluster_name: "outbound|80||istio-egressgateway.istio-system.svc.cluster.local"
endpoints {
  locality {
    region: "us-central1"
    zone: "us-central1-a"
  }
  lb_endpoints {
    endpoint {
      address {
        socket_address {
          address: "10.8.2.85"
          port_value: 80
        }
      }
    }
    metadata {
      filter_metadata {
        key: "istio"
        value {
          fields {
            key: "uid"
            value {
              string_value: "kubernetes://istio-egressgateway-7dc5557587-cvljv.istio-system"
            }
          }
        }
      }
    }
  }
  load_balancing_weight {
    value: 1
  }
}
]
[2019-01-28 23:00:25.890][23][debug][upstream] [external/envoy/source/common/upstream/eds.cc:153] EDS hosts or locality weights changed for cluster: outbound|9093||istio-citadel.istio-system.svc.cluster.local current hosts 0 priority 0
[2019-01-28 23:00:25.890][23][trace][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:899] Local locality: 
[2019-01-28 23:00:25.890][23][debug][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:685] initializing secondary cluster outbound|9093||istio-citadel.istio-system.svc.cluster.local completed
[2019-01-28 23:00:25.890][23][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:928] membership update for TLS cluster outbound|9093||istio-citadel.istio-system.svc.cluster.local added 1 removed 0
[2019-01-28 23:00:25.891][23][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:91] cm init: init complete: cluster=outbound|9093||istio-citadel.istio-system.svc.cluster.local primary=0 secondary=4
[2019-01-28 23:00:25.891][23][debug][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:60] gRPC config for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment accepted with 1 resources: [cluster_name: "outbound|9093||istio-citadel.istio-system.svc.cluster.local"
endpoints {
  locality {
    region: "us-central1"
    zone: "us-central1-a"
  }
  lb_endpoints {
    endpoint {
      address {
        socket_address {
          address: "10.8.3.45"
          port_value: 9093
        }
      }
    }
    metadata {
      filter_metadata {
        key: "istio"
        value {
          fields {
            key: "uid"
            value {
              string_value: "kubernetes://istio-citadel-7bbd9df944-r5pmt.istio-system"
            }
          }
        }
      }
    }
  }
  load_balancing_weight {
    value: 1
  }
}
]
[2019-01-28 23:00:25.891][23][debug][upstream] [external/envoy/source/common/upstream/eds.cc:153] EDS hosts or locality weights changed for cluster: outbound|8060||istio-citadel.istio-system.svc.cluster.local current hosts 0 priority 0
[2019-01-28 23:00:25.891][23][trace][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:899] Local locality: 
[2019-01-28 23:00:25.891][23][debug][upstream] [external/envoy/source/common/upstream/upstream_impl.cc:685] initializing secondary cluster outbound|8060||istio-citadel.istio-system.svc.cluster.local completed
[2019-01-28 23:00:25.891][23][debug][upstream] [external/envoy/source/common/config/grpc_mux_impl.cc:137] gRPC mux subscribe for type.googleapis.com/envoy.api.v2.auth.Secret
[2019-01-28 23:00:25.891][23][trace][upstream] [external/envoy/source/common/config/grpc_mux_impl.cc:63] draining discovery requests 1
[2019-01-28 23:00:25.891][23][debug][upstream] [external/envoy/source/common/config/grpc_mux_impl.cc:89] No stream available to sendDiscoveryRequest for type.googleapis.com/envoy.api.v2.auth.Secret
[2019-01-28 23:00:25.892][23][debug][upstream] [external/envoy/source/common/config/grpc_mux_impl.cc:48] Establishing new gRPC bidi stream for rpc StreamSecrets(stream .envoy.api.v2.DiscoveryRequest) returns (stream .envoy.api.v2.DiscoveryResponse);

[2019-01-28 23:00:25.892][23][trace][upstream] [external/envoy/source/common/config/grpc_mux_impl.cc:63] draining discovery requests 1
[2019-01-28 23:00:25.892][23][trace][upstream] [external/envoy/source/common/config/grpc_mux_impl.cc:114] Sending DiscoveryRequest for type.googleapis.com/envoy.api.v2.auth.Secret: node {
  id: "router~10.8.2.83~istio-ingressgateway-6bd597dc46-sfc4t.istio-system~istio-system.svc.cluster.local"
  cluster: "istio-ingressgateway"
  metadata {
    fields {
      key: "CONFIG_NAMESPACE"
      value {
        string_value: "istio-system"
      }
    }
    fields {
      key: "ISTIO_META_INSTANCE_IPS"
      value {
        string_value: "10.8.2.83,10.8.2.83,fe80::3017:e8ff:feab:f9a8"
      }
    }
    fields {
      key: "ISTIO_PROXY_SHA"
      value {
        string_value: "istio-proxy:7738fa3d506d3a1a7be002c1f06377e6516bc245"
      }
    }
    fields {
      key: "ISTIO_PROXY_VERSION"
      value {
        string_value: "1.1.0"
      }
    }
    fields {
      key: "ISTIO_VERSION"
      value {
        string_value: "1.0-dev"
      }
    }
    fields {
      key: "POD_NAME"
      value {
        string_value: "istio-ingressgateway-6bd597dc46-sfc4t"
      }
    }
    fields {
      key: "ROUTER_MODE"
      value {
        string_value: "sni-dnat"
      }
    }
    fields {
      key: "istio"
      value {
        string_value: "sidecar"
      }
    }
  }
  locality {
  }
  build_version: "7738fa3d506d3a1a7be002c1f06377e6516bc245/1.10.0-dev/Clean/RELEASE/BoringSSL"
}
resource_names: "default"
type_url: "type.googleapis.com/envoy.api.v2.auth.Secret"

[2019-01-28 23:00:25.892][23][trace][grpc] [external/envoy/source/common/grpc/google_async_client_impl.cc:197] Queued message to write (609 bytes)

server side logs -

2019-01-28T22:59:54.629905Z	info	*******received StreamSecrets

2019-01-28T22:59:54.629993Z	info	*******token "eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJiYXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiZGVmYXVsdC10b2tlbi1wNmdsOCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNWE2NTk0NDItMjM1MC0xMWU5LTg3MjQtNDIwMTBhODAwMmJhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmJhcjpkZWZhdWx0In0.pqZjvd3a33jkgpGehLLhM-U13OhZmlkGrujDWt_fbvpwhk4isCfjKntLjsUTY1ZRCTZBdlY1K77Uf4xR5V8HQdWkv6Eo0u82HBAFUaCdTciqbx6IaxHd9siWAlJlB1HAIOs7AALMU2lSjB1CVC-FTKyXqgxE1cqfPAxdrf5bQj5AT-ESYE2t7CjdUtrV8gJLwPEveGoBpafRqN-FoOserX7NDTwyhQfTF_TKsnPPBiLNAk5gXrkdYTu8IpW-AHakKwmWs_k4HEp3Gy23RFn0VVzR-413nvQ06krww3SGnTfbMI5mxhMw5_JW06LLyMt1XEfAY3lLBBJj1qPx6h_WgA"

2019-01-28T22:59:54.631478Z	info	*******received StreamSecrets

2019-01-28T22:59:54.631522Z	info	*******token "eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJiYXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiZGVmYXVsdC10b2tlbi1wNmdsOCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNWE2NTk0NDItMjM1MC0xMWU5LTg3MjQtNDIwMTBhODAwMmJhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmJhcjpkZWZhdWx0In0.pqZjvd3a33jkgpGehLLhM-U13OhZmlkGrujDWt_fbvpwhk4isCfjKntLjsUTY1ZRCTZBdlY1K77Uf4xR5V8HQdWkv6Eo0u82HBAFUaCdTciqbx6IaxHd9siWAlJlB1HAIOs7AALMU2lSjB1CVC-FTKyXqgxE1cqfPAxdrf5bQj5AT-ESYE2t7CjdUtrV8gJLwPEveGoBpafRqN-FoOserX7NDTwyhQfTF_TKsnPPBiLNAk5gXrkdYTu8IpW-AHakKwmWs_k4HEp3Gy23RFn0VVzR-413nvQ06krww3SGnTfbMI5mxhMw5_JW06LLyMt1XEfAY3lLBBJj1qPx6h_WgA"

2019-01-28T22:59:54.634379Z	info	*******received StreamSecrets

2019-01-28T22:59:54.634687Z	info	*******token "eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJiYXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiZGVmYXVsdC10b2tlbi1wNmdsOCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNWE2NTk0NDItMjM1MC0xMWU5LTg3MjQtNDIwMTBhODAwMmJhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmJhcjpkZWZhdWx0In0.pqZjvd3a33jkgpGehLLhM-U13OhZmlkGrujDWt_fbvpwhk4isCfjKntLjsUTY1ZRCTZBdlY1K77Uf4xR5V8HQdWkv6Eo0u82HBAFUaCdTciqbx6IaxHd9siWAlJlB1HAIOs7AALMU2lSjB1CVC-FTKyXqgxE1cqfPAxdrf5bQj5AT-ESYE2t7CjdUtrV8gJLwPEveGoBpafRqN-FoOserX7NDTwyhQfTF_TKsnPPBiLNAk5gXrkdYTu8IpW-AHakKwmWs_k4HEp3Gy23RFn0VVzR-413nvQ06krww3SGnTfbMI5mxhMw5_JW06LLyMt1XEfAY3lLBBJj1qPx6h_WgA"

2019-01-28T22:59:54.635054Z	info	***request proxyID "sidecar~10.8.1.36~httpbin-5676f44967-dpvbv.bar~bar.svc.cluster.local", versionInfo "", resourcename "ROOTCA"

2019-01-28T22:59:54.635597Z	info	***request proxyID "sidecar~10.8.1.36~httpbin-5676f44967-dpvbv.bar~bar.svc.cluster.local", versionInfo "", resourcename "ROOTCA"

2019-01-28T22:59:54.635798Z	info	***request proxyID "sidecar~10.8.1.36~httpbin-5676f44967-dpvbv.bar~bar.svc.cluster.local", versionInfo "", resourcename "default"

2019-01-28T22:59:54.643131Z	info	*******received StreamSecrets

2019-01-28T22:59:54.643205Z	info	*******token "eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJiYXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiZGVmYXVsdC10b2tlbi1wNmdsOCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNWE2NTk0NDItMjM1MC0xMWU5LTg3MjQtNDIwMTBhODAwMmJhIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmJhcjpkZWZhdWx0In0.pqZjvd3a33jkgpGehLLhM-U13OhZmlkGrujDWt_fbvpwhk4isCfjKntLjsUTY1ZRCTZBdlY1K77Uf4xR5V8HQdWkv6Eo0u82HBAFUaCdTciqbx6IaxHd9siWAlJlB1HAIOs7AALMU2lSjB1CVC-FTKyXqgxE1cqfPAxdrf5bQj5AT-ESYE2t7CjdUtrV8gJLwPEveGoBpafRqN-FoOserX7NDTwyhQfTF_TKsnPPBiLNAk5gXrkdYTu8IpW-AHakKwmWs_k4HEp3Gy23RFn0VVzR-413nvQ06krww3SGnTfbMI5mxhMw5_JW06LLyMt1XEfAY3lLBBJj1qPx6h_WgA"

2019-01-28T22:59:54.643268Z	info	***request proxyID "sidecar~10.8.1.36~httpbin-5676f44967-dpvbv.bar~bar.svc.cluster.local", versionInfo "", resourcename "default"

The text was updated successfully, but these errors were encountered:

mattklein123 · 2019-01-29T02:23:14Z

@qiwzhang

qiwzhang · 2019-01-29T02:31:36Z

@JimmyCYJ Could you help?

qiwzhang · 2019-01-30T18:46:12Z

I looked at envoy log, could not figure out the root cause. There were 9 clusters, each with two secrets to fetch {default, RootCA}. But somehow Envoy send two set of requests to SDS server. It should only send one set.

Need to add more debug trace to get more insight.

quanjielin · 2019-02-01T02:41:30Z

I added some logs(istio@44d02ab) and re-run.

envoy config
ingress_config.log, all sds config are in dynamic clusters with two unique resource names
envoy logs
ingress.txt, for one source name and sdsconfig, secret provider was added twice almost at same time, is it expected or racing condition ?

[2019-02-01 02:13:36.437][13][debug][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:57] cds: add/update cluster 'outbound|443||kubernetes.default.svc.cluster.local'
[2019-02-01 02:13:36.440][13][info][secret] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/secret/_virtual_includes/secret_manager_impl_lib/common/secret/secret_manager_impl.h:55] ***config name default
[2019-02-01 02:13:36.440][13][info][secret] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/secret/_virtual_includes/secret_manager_impl_lib/common/secret/secret_manager_impl.h:56] ***sds_config_source api_config_source {
  api_type: GRPC
  grpc_services {
    google_grpc {
      target_uri: "unix:/var/run/sds/uds_path"
      channel_credentials {
        local_credentials {
        }
      }
      call_credentials {
        from_plugin {
          name: "envoy.grpc_credentials.file_based_metadata"
          config {
            fields {
              key: "header_key"
              value {
                string_value: "istio_sds_credentail_header-bin"
              }
            }
            fields {
              key: "secret_data"
              value {
                struct_value {
                  fields {
                    key: "filename"
                    value {
                      string_value: "/var/run/secrets/kubernetes.io/serviceaccount/token"
                    }
                  }
                }
              }
            }
          }
        }
      }
      stat_prefix: "sdsstat"
      credentials_factory_name: "envoy.grpc_credentials.file_based_metadata"
    }
  }
}

[2019-02-01 02:13:36.440][13][info][secret] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/secret/_virtual_includes/secret_manager_impl_lib/common/secret/secret_manager_impl.h:68] ***add secret provider for default
...
[2019-02-01 02:13:36.453][13][debug][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:57] cds: add/update cluster 'outbound|443||istio-egressgateway.istio-system.svc.cluster.local'
[2019-02-01 02:13:36.456][13][info][secret] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/secret/_virtual_includes/secret_manager_impl_lib/common/secret/secret_manager_impl.h:55] ***config name default
[2019-02-01 02:13:36.456][13][info][secret] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/secret/_virtual_includes/secret_manager_impl_lib/common/secret/secret_manager_impl.h:56] ***sds_config_source api_config_source {
  api_type: GRPC
  grpc_services {
    google_grpc {
      target_uri: "unix:/var/run/sds/uds_path"
      channel_credentials {
        local_credentials {
        }
      }
      call_credentials {
        from_plugin {
          name: "envoy.grpc_credentials.file_based_metadata"
          config {
            fields {
              key: "header_key"
              value {
                string_value: "istio_sds_credentail_header-bin"
              }
            }
            fields {
              key: "secret_data"
              value {
                struct_value {
                  fields {
                    key: "filename"
                    value {
                      string_value: "/var/run/secrets/kubernetes.io/serviceaccount/token"
                    }
                  }
                }
              }
            }
          }
        }
      }
      stat_prefix: "sdsstat"
      credentials_factory_name: "envoy.grpc_credentials.file_based_metadata"
    }
  }
}

[2019-02-01 02:13:36.456][13][info][secret] [bazel-out/k8-fastbuild/bin/external/envoy/source/common/secret/_virtual_includes/secret_manager_impl_lib/common/secret/secret_manager_impl.h:68] ***add secret provider for default

JimmyCYJ · 2019-02-01T04:48:11Z

Thanks for the investigation, but I don't think there will be a race condition in SecretManagerImpl. The cds updates all happen in the same thread. Could you please dump the map_key as well, I would like to check if they are the same.

This PR uses hash func that generates deterministic result for proto. Background - When integrate envoy SDS with Istio, we found envoy sends out multiple requests for same sdsconfig(details in #5744); After some debugging, we found the issue describes in [protocolbuffers/protobuf#5668](protocolbuffers/protobuf#5668). Signed-off-by: Quanjie Lin <quanlin@google.com>

lizan · 2019-02-14T22:11:51Z

The problem still exists after #5814, after more investigation, I deduced this to protocolbuffers/protobuf#5731.

cc @htuch @AndresGuedez for SRDS framework which might be affected too.

lizan · 2019-02-14T22:30:03Z

attached diff with 2 deterministically serialized sds_config_source.

htuch · 2019-02-14T22:31:42Z

@lizan good find, is a mitigation while the protobuf issue is investigated to use Struct?

AndresGuedez · 2019-02-14T22:43:39Z

The problem still exists after #5814, after more investigation, I deduced this to protocolbuffers/protobuf#5731.

cc @htuch @AndresGuedez for SRDS framework which might be affected too.

Thanks for the ping. This issue affects RDS as well.

lizan · 2019-02-14T22:52:39Z

@htuch Yeah I think that would work. If the ConfigSource doesn't contain an Any then it is fine anyway.

…proxy#5814) This PR uses hash func that generates deterministic result for proto. Background - When integrate envoy SDS with Istio, we found envoy sends out multiple requests for same sdsconfig(details in envoyproxy#5744); After some debugging, we found the issue describes in [protocolbuffers/protobuf#5668](protocolbuffers/protobuf#5668). Signed-off-by: Quanjie Lin <quanlin@google.com> Signed-off-by: Fred Douglas <fredlas@google.com>

stale · 2019-03-16T23:08:20Z

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

stale · 2019-03-23T23:34:16Z

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted". Thank you for your contributions.

JimmyCYJ · 2019-09-11T21:39:35Z

In Istio, we observe multiple SDS connections for identical SDS config. istio/istio#17027 use Struct instead of Any when generates call credentials for SDS config.

@lizan could you please reopen this issue?

…ny recursively (#8231) Use TextFormat::Print with SetExpandAny(true) instead of SerializeAsString to calculate hash with Any expanded. Risk Level: Med Testing: CI, regression test Fixes #5744. Signed-off-by: Lizan Zhou <lizan@tetrate.io>

…ny recursively (envoyproxy#8231) Use TextFormat::Print with SetExpandAny(true) instead of SerializeAsString to calculate hash with Any expanded. Risk Level: Med Testing: CI, regression test Fixes envoyproxy#5744. Signed-off-by: Lizan Zhou <lizan@tetrate.io>

…ny recursively (envoyproxy#8231) Use TextFormat::Print with SetExpandAny(true) instead of SerializeAsString to calculate hash with Any expanded. Risk Level: Med Testing: CI, regression test Fixes envoyproxy#5744. Signed-off-by: Lizan Zhou <lizan@tetrate.io> (cherry picked from commit 647aea1)

…ny recursively (envoyproxy#8231) Use TextFormat::Print with SetExpandAny(true) instead of SerializeAsString to calculate hash with Any expanded. Risk Level: Med Testing: CI, regression test Fixes envoyproxy#5744. Signed-off-by: Lizan Zhou <lizan@tetrate.io>

…ny recursively (envoyproxy#8231) (#109) Use TextFormat::Print with SetExpandAny(true) instead of SerializeAsString to calculate hash with Any expanded. Risk Level: Med Testing: CI, regression test Fixes envoyproxy#5744. Signed-off-by: Lizan Zhou <lizan@tetrate.io> (cherry picked from commit 647aea1)

mattklein123 added the question Questions that are neither investigations, bugs, nor enhancements label Jan 29, 2019

quanjielin mentioned this issue Feb 2, 2019

use hash func that generates deterministic result for protobuf #5814

Merged

quanjielin mentioned this issue Feb 13, 2019

handle multiple streams in nodeagent istio/istio#11738

Merged

lizan added bug and removed question Questions that are neither investigations, bugs, nor enhancements labels Feb 14, 2019

lizan changed the title ~~why envoy sends two sds request with same sdsconfig resource name~~ SDS/RDS sends two DiscoveryRequest with same config resource name Feb 14, 2019

stale bot added the stale stalebot believes this issue/PR has not been touched recently label Mar 16, 2019

lizan mentioned this issue Mar 21, 2019

config: un-deprecating Structs until the replacement (docs etc.) are fully ready #6346

Merged

stale bot closed this as completed Mar 23, 2019

JimmyCYJ mentioned this issue May 17, 2019

Add a map to manage FileBasedMetadataConfig istio/istio#11753

Merged

JimmyCYJ mentioned this issue Sep 11, 2019

Fallback to struct when generating SDS config istio/istio#17027

Closed

JimmyCYJ mentioned this issue Sep 12, 2019

Pilot should use struct to generate SDS config istio/istio#16974

Closed

lizan reopened this Sep 12, 2019

stale bot removed the stale stalebot believes this issue/PR has not been touched recently label Sep 12, 2019

lizan mentioned this issue Sep 13, 2019

proto: re-implement MessageUtil::hash function to consistently hash Any recursively #8231

Merged

htuch closed this as completed in #8231 Sep 20, 2019

nrjpoddar mentioned this issue Oct 3, 2019

Cherrypick 647aea1 from envoyproxy/envoy istio/envoy#109

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SDS/RDS sends two DiscoveryRequest with same config resource name #5744

SDS/RDS sends two DiscoveryRequest with same config resource name #5744

quanjielin commented Jan 29, 2019 •

edited

Loading

mattklein123 commented Jan 29, 2019

qiwzhang commented Jan 29, 2019

qiwzhang commented Jan 30, 2019

quanjielin commented Feb 1, 2019

JimmyCYJ commented Feb 1, 2019

lizan commented Feb 14, 2019

lizan commented Feb 14, 2019

htuch commented Feb 14, 2019

AndresGuedez commented Feb 14, 2019

lizan commented Feb 14, 2019

stale bot commented Mar 16, 2019

stale bot commented Mar 23, 2019

JimmyCYJ commented Sep 11, 2019 •

edited

Loading

SDS/RDS sends two DiscoveryRequest with same config resource name #5744

SDS/RDS sends two DiscoveryRequest with same config resource name #5744

Comments

quanjielin commented Jan 29, 2019 • edited Loading

mattklein123 commented Jan 29, 2019

qiwzhang commented Jan 29, 2019

qiwzhang commented Jan 30, 2019

quanjielin commented Feb 1, 2019

JimmyCYJ commented Feb 1, 2019

lizan commented Feb 14, 2019

lizan commented Feb 14, 2019

htuch commented Feb 14, 2019

AndresGuedez commented Feb 14, 2019

lizan commented Feb 14, 2019

stale bot commented Mar 16, 2019

stale bot commented Mar 23, 2019

JimmyCYJ commented Sep 11, 2019 • edited Loading

quanjielin commented Jan 29, 2019 •

edited

Loading

JimmyCYJ commented Sep 11, 2019 •

edited

Loading