Empty EDS prevents LDS and RDS requests

[githuberto]

Description:
When connecting to a management server over ADS, Envoy is sending the expected CDS and EDS requests. However, if the EDS response is empty, Envoy is repeatedly requesting EDS and never moving on to LDS and RDS.

Repro steps:

1. Configure an Envoy to use ADS over gRPC
2. Have the management server return a CDS with a cluster but an empty EDS response for that cluster
3. Envoy will keep requesting EDS, never requesting LDS or RDS

Config:

node {
  id: "some_id"
  cluster: "some_envoy_cluste"
  metadata {
    <snipped>
  }
  locality {
    sub_zone: "some_zone"
  }
}
dynamic_resources {
  lds_config {
    ads {
    }
  }
  cds_config {
    ads {
    }
  }
  ads_config {
    api_type: GRPC
    grpc_services {
      google_grpc {
        target_uri: "dns:///example.com:12345"
        stat_prefix: "stats"
        credentials_factory_name: "com.example.creds"
        config {
          fields {
            key: "creds_enabled"
            value {
              bool_value: true
            }
          }
        }
      }
    }
  }
}
cluster_manager {
  load_stats_config {
    api_type: GRPC
    grpc_services {
      google_grpc {
        target_uri: "dns:///example.com:12345"
        stat_prefix: "stats"
        credentials_factory_name: "com.example.creds"
        config {
          fields {
            key: "creds_enabled"
            value {
              bool_value: true
            }
          }
        }
      }
    }
  }
}
stats_sinks {
  name: "com.example.sink"
}
admin {
  access_log_path: "/dev/null"
  address {
    socket_address {
      address: "::1"
      port_value: 43210
    }
  }
}

Logs:

[config] gRPC config for type.googleapis.com/envoy.api.v2.Cluster accepted with 1 resources: [name: "my_eds_cluster"
type: EDS
eds_cluster_config {
  eds_config {
    ads {
    }
  }
}
connect_timeout {
  seconds: 10
}
circuit_breakers {
  thresholds {
    max_connections {
      value: 65536
    }
  }
}
http_protocol_options {
}
outlier_detection {
  interval {
    seconds: 1
  }
  max_ejection_percent {
    value: 50
  }
  enforcing_consecutive_5xx {
  }
  consecutive_gateway_failure {
    value: 3
  }
  enforcing_consecutive_gateway_failure {
    value: 100
  }
}
metadata {
 <snipped>
}
common_lb_config {
  locality_weighted_lb_config {
  }
}
]
[upstream] Sending DiscoveryRequest for type.googleapis.com/envoy.api.v2.Cluster: version_info: "1538789483852555553"
node {
  id: "some_id"
  cluster: "some_envoy_cluster"
  metadata {
    <snipped>
  }
  locality {
    sub_zone: "some_zone"
  }
  build_version: "0/1.8.0-dev/Clean/RELEASE"
}
type_url: "type.googleapis.com/envoy.api.v2.Cluster"
response_nonce: "1"
[upstream] Received gRPC message for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment at version 1
[upstream] Sending DiscoveryRequest for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment: version_info: "1"
node {
  id: "some_id"
  cluster: "some_envoy_cluste"
  metadata {
    <snipped>
  }
  locality {
    sub_zone: "some_zone"
  }
  build_version: "0/1.8.0-dev/Clean/RELEASE"
}
resource_names: "my_eds_cluster"
type_url: "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment"
response_nonce: "1"
[upstream] Load report locality count 0
[upstream] Sending LoadStatsRequest: node {
  id: "some_id"
  cluster: "some_envoy_cluste"
  metadata {
    <snipped>
  }
  locality {
    sub_zone: "some_zone"
  }
  build_version: "0/1.8.0-dev/Clean/RELEASE"
}
cluster_stats {
  cluster_name: "my_eds_cluster"
  load_report_interval {
    seconds: 10
    nanos: 1604000
  }
}
[upstream] New load report epoch: clusters: "my_eds_cluster"
load_reporting_interval {
  seconds: 10
}
[upstream] Received gRPC message for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment at version 2
[upstream] Sending DiscoveryRequest for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment: version_info: "2"
node {
  id: "some_id"
  cluster: "some_envoy_cluster"
  metadata {
    <snipped>
  }
  locality {
    sub_zone: "some_zone"
  }
  build_version: "0/1.8.0-dev/Clean/RELEASE"
}
resource_names: "my_eds_cluster"
type_url: "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment"
response_nonce: "2"
[upstream] Load report locality count 0
[upstream] Sending LoadStatsRequest: node {
  id: "some_id"
  cluster: "some_envoy_cluster"
  metadata {
    <snipped>
  }
  locality {
    sub_zone: "some_zone"
  }
  build_version: "0/1.8.0-dev/Clean/RELEASE"
}
cluster_stats {
  cluster_name: "my_eds_cluster"
  load_report_interval {
    seconds: 10
    nanos: 110620000
  }
}
[upstream] Received gRPC message for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment at version 3
[upstream] Sending DiscoveryRequest for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment: version_info: "3"
node {
  id: "some_id"
  cluster: "some_envoy_cluster"
  metadata {
    <snipped>
  }
  locality {
    sub_zone: "some_zone"
  }
  build_version: "0/1.8.0-dev/Clean/RELEASE"
}
resource_names: "my_eds_cluster"
type_url: "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment"
response_nonce: "3"
[upstream] New load report epoch: clusters: "my_eds_cluster"
load_reporting_interval {
  seconds: 10
}

[htuch]

@ramaraochavali I wonder if #4276 might have led to this issue? Essentially what is happening AFAICT is that cluster initialization never completes, since an empty EDS response is not sufficient to provide an empty initial update to the cluster.

[ramaraochavali]

@htuch I think you are right. I will revert it for now to get back the old behaviour and fix the empty stat issue later