xDS mulitple cluster - gRPC Client stops RPC, when a Cluster has no active endpoint

[axkum10]

Hello,

I am trying to test Weighted Cluster load balancing. Here is problem (in my mind) and scenario/setup I tried:

Setup: xDS server with a Listener -> a Route -> Multiple Cluster -> Endpoint attached to each cluster.

Listener -> Route -> Cluster 1 (weight: 1) --> 2 Endpoints (Server1, Server2)
                  -> Cluster 2 (weight 2)  --> 1 Endpoint (Server4)

Scenario:
Everything good:

1. xDS server starts with above configuration of endpoints.
2. gRPC Client starts:
   a. Connects with xDS server
   b. sends LDS, RDS, CDS and EDS. Gets 3 endpoints
   c. Starts calling all 3 (Server1, Server2, Server4) endpoints (round robin) - all good.

Failure:

1. Server4, which is the only endpoint under Cluster 2, crashes. Note: Now Cluster 2 has no available endpoints.
2. gRPC Client stops calling all available endpoints (Server1 and Server2, listed under Cluster 1 are available).
3. gRPC Client keeps on retrying "CONNECTING" to Server4

The gRPC Client has 2 endpoints available. Shouldn't gRPC Client continue routing traffic to available endpoints.

Example code

Software version:

go 1.22.7
google.golang.org/grpc v1.71.0
google.golang.org/protobuf v1.36.4
github.com/envoyproxy/go-control-plane/envoy v1.32.4

Client Bootstrap file

{
  "xds_servers": [
    {
      "server_uri": "xds.domain.com:8001",
      "channel_creds": [
        {
          "type": "insecure"
        }
      ]
    }
  ],
  "server_features": ["xds_v3"],
  "authorities": {
    "non-default-auth": {
      "xds_servers": [
        {
          "server_uri": "10.157.132.92:8001",
          "channel_creds": [
            {
              "type": "insecure"
            }
          ]
        }
      ]
    }
   },

  "node": {
    "id": "FileNode1",
    "metadata": {
      "R_GCP_PROJECT_NUMBER": "123456789012"
    }
  }
}

Log:

2025/06/05 17:11:03 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Child pickers: map[cluster:localCluster1:picker:0xc0003266a8,state:READY,stateToAggregate:READY cluster:xdtspCluster2:picker:0xc000326918,state:READY,stateToAggregate:READY]
2025/06/05 17:11:03 INFO: [xds] [priority-lb 0xc0004ba770] Switching to ("priority-0-0", 0) in syncPriority
2025/06/05 17:11:03 Client: Received response- 0 message:"Hello request from client  ----> from CLUSTER2_SERVER4"
2025/06/05 17:11:05 Client: Received response- 1 message:"Hello request from client  ----> from CLUSTER1_SERVER2"
2025/06/05 17:11:07 Client: Received response- 2 message:"Hello request from client  ----> from CLUSTER1_SERVER1"
2025/06/05 17:11:09 Client: Received response- 3 message:"Hello request from client  ----> from CLUSTER1_SERVER2"
2025/06/05 17:11:11 Client: Received response- 4 message:"Hello request from client  ----> from CLUSTER1_SERVER1"
2025/06/05 17:11:13 Client: Received response- 5 message:"Hello request from client  ----> from CLUSTER2_SERVER4"
2025/06/05 17:11:15 Client: Received response- 6 message:"Hello request from client  ----> from CLUSTER1_SERVER2"
2025/06/05 17:11:17 Client: Received response- 7 message:"Hello request from client  ----> from CLUSTER2_SERVER4"
2025/06/05 17:11:19 Client: Received response- 8 message:"Hello request from client  ----> from CLUSTER1_SERVER1"
2025/06/05 17:11:21 Client: Received response- 9 message:"Hello request from client  ----> from CLUSTER1_SERVER2"
2025/06/05 17:11:23 Client: Received response- 10 message:"Hello request from client  ----> from CLUSTER1_SERVER1"
2025/06/05 17:11:25 Client: Received response- 11 message:"Hello request from client  ----> from CLUSTER1_SERVER2"
2025/06/05 17:11:26 INFO: [transport] [client-transport 0xc0003e0fc8] Closing: connection error: desc = "error reading from server: EOF"
2025/06/05 17:11:26 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to IDLE
2025/06/05 17:11:26 INFO: [transport] [client-transport 0xc0003e0fc8] loopyWriter exiting with error: connection error: desc = "error reading from server: EOF"
2025/06/05 17:11:26 INFO: [xds] [weighted-target-lb 0xc0000a5100] Balancer state update from child {"region":"useast","zone":"az1","subZone":"sz1"}, new state: {ConnectivityState:IDLE Picker:0xc00001f400}
2025/06/05 17:11:26 INFO: [xds] [weighted-target-lb 0xc0000a5100] Child pickers with config: map[{"region":"useast","zone":"az1","subZone":"sz1"}:weight:1,picker:0xc00001f400,state:IDLE,stateToAggregate:IDLE]
2025/06/05 17:11:26 INFO: [xds] [priority-lb 0xc0004ba770] Balancer state update from child priority-0-0, new state: {ConnectivityState:IDLE Picker:0xc0003fa018}
2025/06/05 17:11:26 INFO: [xds] [priority-lb 0xc0004ba770] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/06/05 17:11:26 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Balancer state update from child cluster:xdtspCluster2, new state: {ConnectivityState:IDLE Picker:0xc0003fa018}
2025/06/05 17:11:26 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] State update from sub-balancer "cluster:xdtspCluster2": {ConnectivityState:IDLE Picker:0xc0003fa018}
2025/06/05 17:11:26 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Child pickers: map[cluster:localCluster1:picker:0xc0003266a8,state:READY,stateToAggregate:READY cluster:xdtspCluster2:picker:0xc0003fa018,state:IDLE,stateToAggregate:IDLE]
2025/06/05 17:11:26 INFO: [xds] [priority-lb 0xc0004ba770] Switching to ("priority-0-0", 0) in syncPriority
2025/06/05 17:11:26 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to CONNECTING
2025/06/05 17:11:26 INFO: [core] [Channel #1 SubChannel #9]Subchannel picks a new address "10.157.145.234:50057" to connect
2025/06/05 17:11:26 INFO: [xds] [weighted-target-lb 0xc0000a5100] Balancer state update from child {"region":"useast","zone":"az1","subZone":"sz1"}, new state: {ConnectivityState:CONNECTING Picker:0xc00001f440}
2025/06/05 17:11:26 INFO: [xds] [weighted-target-lb 0xc0000a5100] Child pickers with config: map[{"region":"useast","zone":"az1","subZone":"sz1"}:weight:1,picker:0xc00001f440,state:CONNECTING,stateToAggregate:CONNECTING]
2025/06/05 17:11:26 INFO: [xds] [priority-lb 0xc0004ba770] Balancer state update from child priority-0-0, new state: {ConnectivityState:CONNECTING Picker:0xc0004d0078}
2025/06/05 17:11:26 INFO: [xds] [priority-lb 0xc0004ba770] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/06/05 17:11:26 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Balancer state update from child cluster:xdtspCluster2, new state: {ConnectivityState:CONNECTING Picker:0xc0004d0078}
2025/06/05 17:11:26 INFO: [core] Creating new client transport to "{Addr: \"10.157.145.234:50057\", ServerName: \"xdstp.upstream.xdspoc.com\", Attributes: {\"<%!p(xds.handshakeClusterNameKey={})>\": \"xdtspCluster2\" , \"<%!p(xds.handshakeAttrKey={})>\": \"<0xc000088f78>\" }, BalancerAttributes: {\"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>\": \"<0xc0000a4aa0>\" , \"<%!p(wrrlocality.attributeKey={})>\": \"Locality Weight: 1\" , \"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>\": \"<%!p(internal.LocalityID={useast az1 sz1})>\" , \"<%!p(weightedroundrobin.attributeKey={})>\": \"Weight: 1\" , \"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" }}": connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:26 WARNING: [core] [Channel #1 SubChannel #9]grpc: addrConn.createTransport failed to connect to {Addr: "10.157.145.234:50057", ServerName: "xdstp.upstream.xdspoc.com", Attributes: {"<%!p(xds.handshakeClusterNameKey={})>": "xdtspCluster2" , "<%!p(xds.handshakeAttrKey={})>": "<0xc000088f78>" }, BalancerAttributes: {"<%!p(weightedroundrobin.attributeKey={})>": "Weight: 1" , "<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" , "<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>": "<0xc0000a4aa0>" , "<%!p(wrrlocality.attributeKey={})>": "Locality Weight: 1" , "<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>": "<%!p(internal.LocalityID={useast az1 sz1})>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:26 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:26 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] State update from sub-balancer "cluster:xdtspCluster2": {ConnectivityState:CONNECTING Picker:0xc0004d0078}
2025/06/05 17:11:26 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Child pickers: map[cluster:localCluster1:picker:0xc0003266a8,state:READY,stateToAggregate:READY cluster:xdtspCluster2:picker:0xc0004d0078,state:CONNECTING,stateToAggregate:CONNECTING]
2025/06/05 17:11:26 INFO: [xds] [priority-lb 0xc0004ba770] Switching to ("priority-0-0", 0) in syncPriority
2025/06/05 17:11:26 INFO: [xds] [weighted-target-lb 0xc0000a5100] Balancer state update from child {"region":"useast","zone":"az1","subZone":"sz1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003d66c0}
2025/06/05 17:11:26 INFO: [xds] [weighted-target-lb 0xc0000a5100] Child pickers with config: map[{"region":"useast","zone":"az1","subZone":"sz1"}:weight:1,picker:0xc0003d66c0,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:26 INFO: [xds] [priority-lb 0xc0004ba770] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003fa150}
2025/06/05 17:11:26 INFO: [xds] [priority-lb 0xc0004ba770] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/06/05 17:11:26 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Balancer state update from child cluster:xdtspCluster2, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003fa150}
2025/06/05 17:11:26 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] State update from sub-balancer "cluster:xdtspCluster2": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003fa150}
2025/06/05 17:11:26 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Child pickers: map[cluster:localCluster1:picker:0xc0003266a8,state:READY,stateToAggregate:READY cluster:xdtspCluster2:picker:0xc0003fa150,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:26 INFO: [xds] [priority-lb 0xc0004ba770] Switching to ("priority-0-0", 0) in syncPriority
2025/06/05 17:11:27 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:27 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to CONNECTING
2025/06/05 17:11:27 INFO: [core] [Channel #1 SubChannel #9]Subchannel picks a new address "10.157.145.234:50057" to connect
2025/06/05 17:11:27 INFO: [core] Creating new client transport to "{Addr: \"10.157.145.234:50057\", ServerName: \"xdstp.upstream.xdspoc.com\", Attributes: {\"<%!p(xds.handshakeClusterNameKey={})>\": \"xdtspCluster2\" , \"<%!p(xds.handshakeAttrKey={})>\": \"<0xc000088f78>\" }, BalancerAttributes: {\"<%!p(wrrlocality.attributeKey={})>\": \"Locality Weight: 1\" , \"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>\": \"<%!p(internal.LocalityID={useast az1 sz1})>\" , \"<%!p(weightedroundrobin.attributeKey={})>\": \"Weight: 1\" , \"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" , \"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>\": \"<0xc0000a4aa0>\" }}": connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:27 WARNING: [core] [Channel #1 SubChannel #9]grpc: addrConn.createTransport failed to connect to {Addr: "10.157.145.234:50057", ServerName: "xdstp.upstream.xdspoc.com", Attributes: {"<%!p(xds.handshakeClusterNameKey={})>": "xdtspCluster2" , "<%!p(xds.handshakeAttrKey={})>": "<0xc000088f78>" }, BalancerAttributes: {"<%!p(wrrlocality.attributeKey={})>": "Locality Weight: 1" , "<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>": "<%!p(internal.LocalityID={useast az1 sz1})>" , "<%!p(weightedroundrobin.attributeKey={})>": "Weight: 1" , "<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" , "<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>": "<0xc0000a4aa0>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:27 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:27 INFO: [xds] [weighted-target-lb 0xc0000a5100] Balancer state update from child {"region":"useast","zone":"az1","subZone":"sz1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003d6780}
2025/06/05 17:11:27 INFO: [xds] [weighted-target-lb 0xc0000a5100] Child pickers with config: map[{"region":"useast","zone":"az1","subZone":"sz1"}:weight:1,picker:0xc0003d6780,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:27 INFO: [xds] [priority-lb 0xc0004ba770] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003fa270}
2025/06/05 17:11:27 INFO: [xds] [priority-lb 0xc0004ba770] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/06/05 17:11:27 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Balancer state update from child cluster:xdtspCluster2, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003fa270}
2025/06/05 17:11:27 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] State update from sub-balancer "cluster:xdtspCluster2": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003fa270}
2025/06/05 17:11:27 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Child pickers: map[cluster:localCluster1:picker:0xc0003266a8,state:READY,stateToAggregate:READY cluster:xdtspCluster2:picker:0xc0003fa270,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:27 INFO: [xds] [priority-lb 0xc0004ba770] Switching to ("priority-0-0", 0) in syncPriority
2025/06/05 17:11:29 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:29 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to CONNECTING
2025/06/05 17:11:29 INFO: [core] [Channel #1 SubChannel #9]Subchannel picks a new address "10.157.145.234:50057" to connect
2025/06/05 17:11:29 INFO: [core] Creating new client transport to "{Addr: \"10.157.145.234:50057\", ServerName: \"xdstp.upstream.xdspoc.com\", Attributes: {\"<%!p(xds.handshakeClusterNameKey={})>\": \"xdtspCluster2\" , \"<%!p(xds.handshakeAttrKey={})>\": \"<0xc000088f78>\" }, BalancerAttributes: {\"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" , \"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>\": \"<0xc0000a4aa0>\" , \"<%!p(wrrlocality.attributeKey={})>\": \"Locality Weight: 1\" , \"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>\": \"<%!p(internal.LocalityID={useast az1 sz1})>\" , \"<%!p(weightedroundrobin.attributeKey={})>\": \"Weight: 1\" }}": connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:29 WARNING: [core] [Channel #1 SubChannel #9]grpc: addrConn.createTransport failed to connect to {Addr: "10.157.145.234:50057", ServerName: "xdstp.upstream.xdspoc.com", Attributes: {"<%!p(xds.handshakeClusterNameKey={})>": "xdtspCluster2" , "<%!p(xds.handshakeAttrKey={})>": "<0xc000088f78>" }, BalancerAttributes: {"<%!p(weightedroundrobin.attributeKey={})>": "Weight: 1" , "<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" , "<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>": "<0xc0000a4aa0>" , "<%!p(wrrlocality.attributeKey={})>": "Locality Weight: 1" , "<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>": "<%!p(internal.LocalityID={useast az1 sz1})>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:29 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:29 INFO: [xds] [weighted-target-lb 0xc0000a5100] Balancer state update from child {"region":"useast","zone":"az1","subZone":"sz1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003d6900}
2025/06/05 17:11:29 INFO: [xds] [weighted-target-lb 0xc0000a5100] Child pickers with config: map[{"region":"useast","zone":"az1","subZone":"sz1"}:weight:1,picker:0xc0003d6900,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:29 INFO: [xds] [priority-lb 0xc0004ba770] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003fa3c0}
2025/06/05 17:11:29 INFO: [xds] [priority-lb 0xc0004ba770] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/06/05 17:11:29 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Balancer state update from child cluster:xdtspCluster2, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003fa3c0}
2025/06/05 17:11:29 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] State update from sub-balancer "cluster:xdtspCluster2": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0003fa3c0}
2025/06/05 17:11:29 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Child pickers: map[cluster:localCluster1:picker:0xc0003266a8,state:READY,stateToAggregate:READY cluster:xdtspCluster2:picker:0xc0003fa3c0,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:29 INFO: [xds] [priority-lb 0xc0004ba770] Switching to ("priority-0-0", 0) in syncPriority
2025/06/05 17:11:31 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:31 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to CONNECTING
2025/06/05 17:11:31 INFO: [core] [Channel #1 SubChannel #9]Subchannel picks a new address "10.157.145.234:50057" to connect
2025/06/05 17:11:31 INFO: [core] Creating new client transport to "{Addr: \"10.157.145.234:50057\", ServerName: \"xdstp.upstream.xdspoc.com\", Attributes: {\"<%!p(xds.handshakeClusterNameKey={})>\": \"xdtspCluster2\" , \"<%!p(xds.handshakeAttrKey={})>\": \"<0xc000088f78>\" }, BalancerAttributes: {\"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>\": \"<0xc0000a4aa0>\" , \"<%!p(wrrlocality.attributeKey={})>\": \"Locality Weight: 1\" , \"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>\": \"<%!p(internal.LocalityID={useast az1 sz1})>\" , \"<%!p(weightedroundrobin.attributeKey={})>\": \"Weight: 1\" , \"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" }}": connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:31 WARNING: [core] [Channel #1 SubChannel #9]grpc: addrConn.createTransport failed to connect to {Addr: "10.157.145.234:50057", ServerName: "xdstp.upstream.xdspoc.com", Attributes: {"<%!p(xds.handshakeClusterNameKey={})>": "xdtspCluster2" , "<%!p(xds.handshakeAttrKey={})>": "<0xc000088f78>" }, BalancerAttributes: {"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>": "<0xc0000a4aa0>" , "<%!p(wrrlocality.attributeKey={})>": "Locality Weight: 1" , "<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>": "<%!p(internal.LocalityID={useast az1 sz1})>" , "<%!p(weightedroundrobin.attributeKey={})>": "Weight: 1" , "<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:31 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:31 INFO: [xds] [weighted-target-lb 0xc0000a5100] Balancer state update from child {"region":"useast","zone":"az1","subZone":"sz1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc00001f600}
2025/06/05 17:11:31 INFO: [xds] [weighted-target-lb 0xc0000a5100] Child pickers with config: map[{"region":"useast","zone":"az1","subZone":"sz1"}:weight:1,picker:0xc00001f600,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:31 INFO: [xds] [priority-lb 0xc0004ba770] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004d0180}
2025/06/05 17:11:31 INFO: [xds] [priority-lb 0xc0004ba770] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/06/05 17:11:31 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Balancer state update from child cluster:xdtspCluster2, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004d0180}
2025/06/05 17:11:31 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] State update from sub-balancer "cluster:xdtspCluster2": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004d0180}
2025/06/05 17:11:31 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Child pickers: map[cluster:localCluster1:picker:0xc0003266a8,state:READY,stateToAggregate:READY cluster:xdtspCluster2:picker:0xc0004d0180,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:31 INFO: [xds] [priority-lb 0xc0004ba770] Switching to ("priority-0-0", 0) in syncPriority
2025/06/05 17:11:35 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:35 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to CONNECTING
2025/06/05 17:11:35 INFO: [core] [Channel #1 SubChannel #9]Subchannel picks a new address "10.157.145.234:50057" to connect
2025/06/05 17:11:35 INFO: [core] Creating new client transport to "{Addr: \"10.157.145.234:50057\", ServerName: \"xdstp.upstream.xdspoc.com\", Attributes: {\"<%!p(xds.handshakeClusterNameKey={})>\": \"xdtspCluster2\" , \"<%!p(xds.handshakeAttrKey={})>\": \"<0xc000088f78>\" }, BalancerAttributes: {\"<%!p(wrrlocality.attributeKey={})>\": \"Locality Weight: 1\" , \"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>\": \"<%!p(internal.LocalityID={useast az1 sz1})>\" , \"<%!p(weightedroundrobin.attributeKey={})>\": \"Weight: 1\" , \"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" , \"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>\": \"<0xc0000a4aa0>\" }}": connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:35 WARNING: [core] [Channel #1 SubChannel #9]grpc: addrConn.createTransport failed to connect to {Addr: "10.157.145.234:50057", ServerName: "xdstp.upstream.xdspoc.com", Attributes: {"<%!p(xds.handshakeClusterNameKey={})>": "xdtspCluster2" , "<%!p(xds.handshakeAttrKey={})>": "<0xc000088f78>" }, BalancerAttributes: {"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>": "<0xc0000a4aa0>" , "<%!p(wrrlocality.attributeKey={})>": "Locality Weight: 1" , "<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>": "<%!p(internal.LocalityID={useast az1 sz1})>" , "<%!p(weightedroundrobin.attributeKey={})>": "Weight: 1" , "<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:35 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:35 INFO: [xds] [weighted-target-lb 0xc0000a5100] Balancer state update from child {"region":"useast","zone":"az1","subZone":"sz1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc00001f6c0}
2025/06/05 17:11:35 INFO: [xds] [weighted-target-lb 0xc0000a5100] Child pickers with config: map[{"region":"useast","zone":"az1","subZone":"sz1"}:weight:1,picker:0xc00001f6c0,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:35 INFO: [xds] [priority-lb 0xc0004ba770] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004d0288}
2025/06/05 17:11:35 INFO: [xds] [priority-lb 0xc0004ba770] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/06/05 17:11:35 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Balancer state update from child cluster:xdtspCluster2, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004d0288}
2025/06/05 17:11:35 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] State update from sub-balancer "cluster:xdtspCluster2": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004d0288}
2025/06/05 17:11:35 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Child pickers: map[cluster:localCluster1:picker:0xc0003266a8,state:READY,stateToAggregate:READY cluster:xdtspCluster2:picker:0xc0004d0288,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:35 INFO: [xds] [priority-lb 0xc0004ba770] Switching to ("priority-0-0", 0) in syncPriority



2025/06/05 17:11:41 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:41 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to CONNECTING
2025/06/05 17:11:41 INFO: [core] [Channel #1 SubChannel #9]Subchannel picks a new address "10.157.145.234:50057" to connect
2025/06/05 17:11:41 INFO: [core] Creating new client transport to "{Addr: \"10.157.145.234:50057\", ServerName: \"xdstp.upstream.xdspoc.com\", Attributes: {\"<%!p(xds.handshakeClusterNameKey={})>\": \"xdtspCluster2\" , \"<%!p(xds.handshakeAttrKey={})>\": \"<0xc000088f78>\" }, BalancerAttributes: {\"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>\": \"<0xc0000a4aa0>\" , \"<%!p(wrrlocality.attributeKey={})>\": \"Locality Weight: 1\" , \"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>\": \"<%!p(internal.LocalityID={useast az1 sz1})>\" , \"<%!p(weightedroundrobin.attributeKey={})>\": \"Weight: 1\" , \"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" }}": connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:41 WARNING: [core] [Channel #1 SubChannel #9]grpc: addrConn.createTransport failed to connect to {Addr: "10.157.145.234:50057", ServerName: "xdstp.upstream.xdspoc.com", Attributes: {"<%!p(xds.handshakeClusterNameKey={})>": "xdtspCluster2" , "<%!p(xds.handshakeAttrKey={})>": "<0xc000088f78>" }, BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" , "<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>": "<0xc0000a4aa0>" , "<%!p(wrrlocality.attributeKey={})>": "Locality Weight: 1" , "<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>": "<%!p(internal.LocalityID={useast az1 sz1})>" , "<%!p(weightedroundrobin.attributeKey={})>": "Weight: 1" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:41 INFO: [core] [Channel #1 SubChannel #9]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 10.157.145.234:50057: connect: connection refused"
2025/06/05 17:11:41 INFO: [xds] [weighted-target-lb 0xc0000a5100] Balancer state update from child {"region":"useast","zone":"az1","subZone":"sz1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc00001f780}
2025/06/05 17:11:41 INFO: [xds] [weighted-target-lb 0xc0000a5100] Child pickers with config: map[{"region":"useast","zone":"az1","subZone":"sz1"}:weight:1,picker:0xc00001f780,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:41 INFO: [xds] [priority-lb 0xc0004ba770] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004d0c00}
2025/06/05 17:11:41 INFO: [xds] [priority-lb 0xc0004ba770] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/06/05 17:11:41 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Balancer state update from child cluster:xdtspCluster2, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004d0c00}
2025/06/05 17:11:41 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] State update from sub-balancer "cluster:xdtspCluster2": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004d0c00}
2025/06/05 17:11:41 INFO: [xds] [xds-cluster-manager-lb 0xc0002ef620] Child pickers: map[cluster:localCluster1:picker:0xc0003266a8,state:READY,stateToAggregate:READY cluster:xdtspCluster2:picker:0xc0004d0c00,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/06/05 17:11:41 INFO: [xds] [priority-lb 0xc0004ba770] Switching to ("priority-0-0", 0) in syncPriority

[arjan-bal]

I attempted to repro this situation in a unit test by having two clusters with 2 endpoints each. I specified a weighted clusters route action that splits traffic evenly b/w the two clusters. After servers for both the endpoints in cluster 2 were stopped, I noticed that half the requests would still go to cluster 2 and fail, the other half would go to cluster 1 and succeed. This seems to be in accordance with the behaviour specified in the gRFC:

This means two things:

1. connectivity state is not considered when making this routing decision, and
2. once a cluster is chosen for an RPC, retry attempts will only use this cluster. This matches Envoy behavior, and also supersedes the behavior specified in gRFC A28.

Unit test repro: arjan-bal#18

[arjan-bal]

I see the problem, in your test client, you've set grpc.WaitForReady(true). This causes the current RPC call to block until the server is up. You would need to either set a shorter context timeout or remove the WaitForReady call option to allow the call to fail and unblock further requests.

[axkum10]

Thanks @arjan-bal, I changed gRPC client code and it is behaving the way you described about unit test.

the gRFC outcome adds a drawback, some request (going to a cluster with no active upstream) will continue to fail (take hit on availability or latency).

In my opinion choosing to follow Envoy behavior does not fit gRPC capability & intent of using what's available i.e. The gRPC Client is already aware of broken connection and can decide to route request to a good connection.

1. connectivity state is not considered when making this routing decision, and 2. once a cluster is chosen for an RPC, retry attempts will only use this cluster. This matches Envoy behavior, and also supersedes the behavior specified in [gRFC A28](https://github.com/grpc/proposal/blob/master/A28-xds-traffic-splitting-and-routing.md).

Important to note: Taking out an inactive upstream (end point) takes minutes in Envoy. Which makes sense, to avoid chaos.

This particular case, I would very much, not want gRPC xDS to be like Envoy i.e. keep sending request to bad cluster or update xDS cache too frequently and create chaos.

[arjan-bal]

Since the decision to send requests to a cluster with only failing subchannels is a cross-language behaviour, not something specific to grpc-go, asking @markdroth to chime in.

Mark, can you share the reasoning behind the behaviour specified in the gRFC:

This means two things:

1. connectivity state is not considered when making this routing decision, and
2. once a cluster is chosen for an RPC, retry attempts will only use this cluster. This matches Envoy behavior, and also supersedes the behavior specified in gRFC A28.