Multiple xDS server - run time failover does not work

[axkum10]

Hello,

I was testing client failover with multiple xDS server setup (earlier reported defect).

gRPC client is not connecting to other xDS server at runtime.
here is test setup/test scenario:

1. Multiple(2) xDS servers:
   a. xDS server 8001 - Up/running
   b. xDS server 8003 - Up

2. Multiple (2) gRPC servers:
   a. gRPC server1 - Up (registered with xDS server 8003, gRPC server port: 50051)
   b. gRPC server2 - Up (registered with xDS server 8001, gRPC server port: 50053)

3. Client: Bootstrap has xDS 8003 (1st entry) and xDS 8001 (2nd entry)
   a. Client starts - Connected to xDS 8003, gets server1 endpoint and starts communicating (Everything is fine till this point)

Failure Injection:
a. xDS 8003 goes down
Observation:
- Client1 is able to continue communicating to server1, with error due to xDS server 8003 connection failure.
- Client tries to connect to xDS server 8001 (does not look at xDS server 8001)
b. server1 goes down.
Result:
- Client tries to connect to server1
- Client tries to connect to xDS server 8003 (1st entry in bootstrap), does not look at xDS server 8001 (2nd entry in bootstrap file)

I was expecting client to failover to xDS server 8003 (either when xDS server 8001 went down or after gRPC server1 went down).

Client Boostrap file:

{
  "xds_servers": [
    {
      "server_uri": "xds.domain.com:8003",
      "channel_creds": [
        {
          "type": "insecure"
        }
      ],
      "server_features": ["xds_v3"]
    },
    {"server_uri": "xds.domain.com:8001",
      "channel_creds": [
        {
          "type": "insecure"
        }
      ],
      "server_features": ["xds_v3"]
    }
  ],
  "node": {
    "id": "test-foo",
    "metadata": {
      "R_GCP_PROJECT_NUMBER": "123456789012"
    }
  }
}

Log (dividing into 2 part: initial setup and successful connection & failure injection)
Complete log file is attached : Failover_not_working.txt
Initial setup and successful connection:

[user31@test-machine]$ go run src/client.go --host xds:///upstream.host.foo
2025/03/16 14:25:01 INFO: [xds] [xds-bootstrap] Using bootstrap file with name "/home/dc-user/grpc/example/employee/go/dynamic/xds/multiple_xDS_client_xds_bootstrap.json" from GRPC_XDS_BOOTSTRAP environment variable
2025/03/16 14:25:01 CHECK FALL BACK
2025/03/16 14:25:01 FALLBACK:false
2025/03/16 14:25:01 INFO: [core] original dial target is: "xds:///upstream.host.foo"
2025/03/16 14:25:01 INFO: [core] [Channel #1]Channel created
2025/03/16 14:25:01 INFO: [core] [Channel #1]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"xds", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/upstream.host.foo", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/03/16 14:25:01 INFO: [core] [Channel #1]Channel authority set to "upstream.host.foo"
2025/03/16 14:25:01 INFO: [xds] [xds-resolver 0xc000360000] Creating resolver for target: xds:///upstream.host.foo
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] Created client with name "xds:///upstream.host.foo" and bootstrap configuration:
 {
  "xds_servers": [
   {
    "server_uri": "xds.domain.com:8003",
    "channel_creds": [
     {
      "type": "insecure"
     }
    ],
    "server_features": [
     "xds_v3"
    ]
   },
   {
    "server_uri": "xds.domain.com:8001",
    "channel_creds": [
     {
      "type": "insecure"
     }
    ],
    "server_features": [
     "xds_v3"
    ]
   }
  ],
  "client_default_listener_resource_name_template": "%s",
  "node": {
   "id": "test_foo",
   "locality": {},
   "metadata": {
    "R_GCP_PROJECT_NUMBER": "123456789012"
   }
  }
 }
2025/03/16 14:25:01 INFO: [xds] xDS node ID: test-foo
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [authority ""] New watch for type "ListenerResource", resource name "upstream.host.foo"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] Received request for a reference to an xdsChannel for server config "xds.domain.com:8003-insecure-xds_v3"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] Creating a new xdsChannel for server config "xds.domain.com:8003-insecure-xds_v3"
2025/03/16 14:25:01 INFO: [core] original dial target is: "xds.domain.com:8003"
2025/03/16 14:25:01 INFO: [core] [Channel #2]Channel created
2025/03/16 14:25:01 INFO: [core] [Channel #2]parsed dial target is: resolver.Target{URL:url.URL{Scheme:"dns", Opaque:"", User:(*url.Userinfo)(nil), Host:"", Path:"/xds.domain.com:8003", RawPath:"", OmitHost:false, ForceQuery:false, RawQuery:"", Fragment:"", RawFragment:""}}
2025/03/16 14:25:01 INFO: [core] [Channel #2]Channel authority set to "xds.domain.com:8003"
2025/03/16 14:25:01 INFO: [core] [Channel #2]Channel exiting idle mode
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] Adding authority "" to the set of interested authorities for channel [0xc00036e120]
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [authority ""] First watch for type "ListenerResource", resource name "upstream.host.foo"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Subscribing to resource "upstream.host.foo" of type "ListenerResource"
2025/03/16 14:25:01 INFO: [core] [Channel #1]Channel exiting idle mode
2025/03/16 14:25:01 INFO: [core] [Channel #2]Resolver state updated: {
  "Addresses": [
    {
      "Addr": "127.0.0.1:8003",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:8003",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)
2025/03/16 14:25:01 INFO: [core] [Channel #2]Channel switches to new LB policy "pick_first"
2025/03/16 14:25:01 INFO: [pick-first-lb] [pick-first-lb 0xc000052510] Received new config {
  "shuffleAddressList": false
}, resolver state {
  "Addresses": [
    {
      "Addr": "127.0.0.1:8003",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:8003",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
}
2025/03/16 14:25:01 INFO: [core] [Channel #2 SubChannel #3]Subchannel created
2025/03/16 14:25:01 INFO: [core] [Channel #2]Channel Connectivity change to CONNECTING
2025/03/16 14:25:01 INFO: [core] [Channel #2 SubChannel #3]Subchannel Connectivity change to CONNECTING
2025/03/16 14:25:01 INFO: [core] [Channel #2 SubChannel #3]Subchannel picks a new address "127.0.0.1:8003" to connect
2025/03/16 14:25:01 INFO: [pick-first-lb] [pick-first-lb 0xc000052510] Received SubConn state update: 0xc0000be190, {ConnectivityState:CONNECTING ConnectionError:<nil> connectedAddress:{Addr: ServerName: Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}
2025/03/16 14:25:01 INFO: [core] [Channel #2 SubChannel #3]Subchannel Connectivity change to READY
2025/03/16 14:25:01 INFO: [pick-first-lb] [pick-first-lb 0xc000052510] Received SubConn state update: 0xc0000be190, {ConnectivityState:READY ConnectionError:<nil> connectedAddress:{Addr:127.0.0.1:8003 ServerName:xds.domain.com:8003 Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}
2025/03/16 14:25:01 INFO: [core] [Channel #2]Channel Connectivity change to READY
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS stream created
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS request sent: {
  "node":  {
    "id":  "test_foo",
    "metadata":  {
      "R_GCP_PROJECT_NUMBER":  "123456789012"
    },
    "userAgentName":  "gRPC Go",
    "userAgentVersion":  "1.71.0",
    "clientFeatures":  [
      "envoy.lb.does_not_support_overprovisioning",
      "xds.config.resource-in-sotw"
    ]
  },
  "resourceNames":  [
    "upstream.host.foo"
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.listener.v3.Listener"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS response received: {
  "versionInfo":  "1",
  "resources":  [
    {
      "@type":  "type.googleapis.com/envoy.config.listener.v3.Listener",
      "name":  "upstream.host.foo",
      "apiListener":  {
        "apiListener":  {
          "@type":  "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
          "rds":  {
            "configSource":  {
              "ads":  {},
              "resourceApiVersion":  "V3"
            },
            "routeConfigName":  "route-upstream.host.foo"
          },
          "httpFilters":  [
            {
              "name":  "envoy.filters.http.router",
              "typedConfig":  {
                "@type":  "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
              }
            }
          ]
        }
      }
    }
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.listener.v3.Listener",
  "nonce":  "1"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Sending ACK for resource type: "type.googleapis.com/envoy.config.listener.v3.Listener", version: "1", nonce: "1"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS request sent: {
  "versionInfo":  "1",
  "resourceNames":  [
    "upstream.host.foo"
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.listener.v3.Listener",
  "responseNonce":  "1"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [authority ""] Resource type "ListenerResource" with name "upstream.host.foo" added to cache
2025/03/16 14:25:01 INFO: [xds] [xds-resolver 0xc000360000] Received update for Listener resource "upstream.host.foo": {
  "RouteConfigName": "route-upstream.host.foo",
  "InlineRouteConfig": null,
  "MaxStreamDuration": 0,
  "HTTPFilters": [
    {
      "Name": "envoy.filters.http.router",
      "Filter": {},
      "Config": {
        "FilterConfig": null
      }
    }
  ],
  "InboundListenerCfg": null,
  "Raw": {
    "type_url": "type.googleapis.com/envoy.config.listener.v3.Listener",
    "value": "ChF1cHN0cmVhbS5ob3N0LmZvb5oB8QEK7gEKZXR5cGUuZ29vZ2xlYXBpcy5jb20vZW52b3kuZXh0ZW5zaW9ucy5maWx0ZXJzLm5ldHdvcmsuaHR0cF9jb25uZWN0aW9uX21hbmFnZXIudjMuSHR0cENvbm5lY3Rpb25NYW5hZ2VyEoQBKmEKGWVudm95LmZpbHRlcnMuaHR0cC5yb3V0ZXIiRApCdHlwZS5nb29nbGVhcGlzLmNvbS9lbnZveS5leHRlbnNpb25zLmZpbHRlcnMuaHR0cC5yb3V0ZXIudjMuUm91dGVyGh8KBDACGgASF3JvdXRlLXVwc3RyZWFtLmhvc3QuZm9v"
  }
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [authority ""] New watch for type "RouteConfigResource", resource name "route-upstream.host.foo"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [authority ""] First watch for type "RouteConfigResource", resource name "route-upstream.host.foo"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Subscribing to resource "route-upstream.host.foo" of type "RouteConfigResource"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS request sent: {
  "resourceNames":  [
    "route-upstream.host.foo"
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.route.v3.RouteConfiguration"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS response received: {
  "versionInfo":  "1",
  "resources":  [
    {
      "@type":  "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
      "name":  "route-upstream.host.foo",
      "virtualHosts":  [
        {
          "name":  "vms_echo",
          "domains":  [
            "*"
          ],
          "routes":  [
            {
              "match":  {
                "prefix":  ""
              },
              "route":  {
                "cluster":  "cluster-upstream.host.foo"
              }
            }
          ]
        }
      ],
      "validateClusters":  true
    }
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
  "nonce":  "2"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Sending ACK for resource type: "type.googleapis.com/envoy.config.route.v3.RouteConfiguration", version: "1", nonce: "2"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS request sent: {
  "versionInfo":  "1",
  "resourceNames":  [
    "route-upstream.host.foo"
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
  "responseNonce":  "2"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [authority ""] Resource type "RouteConfigResource" with name "route-upstream.host.foo" added to cache
2025/03/16 14:25:01 INFO: [xds] [xds-resolver 0xc000360000] Received update for RouteConfiguration resource "route-upstream.host.foo": {
  "VirtualHosts": [
    {
      "Domains": [
        "*"
      ],
      "Routes": [
        {
          "Path": null,
          "Prefix": "",
          "Regex": null,
          "CaseInsensitive": false,
          "Headers": null,
          "Fraction": null,
          "HashPolicies": null,
          "MaxStreamDuration": null,
          "HTTPFilterConfigOverride": null,
          "RetryConfig": null,
          "ActionType": 1,
          "WeightedClusters": {
            "cluster-upstream.host.foo": {
              "Weight": 1,
              "HTTPFilterConfigOverride": null
            }
          },
          "ClusterSpecifierPlugin": ""
        }
      ],
      "HTTPFilterConfigOverride": null,
      "RetryConfig": null
    }
  ],
  "ClusterSpecifierPlugins": {},
  "Raw": {
    "type_url": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
    "value": "Chdyb3V0ZS11cHN0cmVhbS5ob3N0LmZvbxIwCgh2bXNfZWNobxIBKhohCgIKABIbChljbHVzdGVyLXVwc3RyZWFtLmhvc3QuZm9vOgIIAQ=="
  }
}
2025/03/16 14:25:01 INFO: [xds] [xds-resolver 0xc000360000] For Listener resource "upstream.host.foo" and RouteConfiguration resource "route-upstream.host.foo", generated service config: {
  "loadBalancingConfig": [
    {
      "xds_cluster_manager_experimental": {
        "children": {
          "cluster:cluster-upstream.host.foo": {
            "childPolicy": [
              {
                "cds_experimental": {
                  "cluster": "cluster-upstream.host.foo"
                }
              }
            ]
          }
        }
      }
    }
  ]
}
2025/03/16 14:25:01 INFO: [core] [Channel #1]Resolver state updated: {
  "Addresses": null,
  "Endpoints": [],
  "ServiceConfig": {
    "Config": {
      "Config": null,
      "Methods": {}
    },
    "Err": null
  },
  "Attributes": {
    "\u003c%!p(xdsclient.clientKeyType=grpc.xds.internal.client.Client)\u003e": "\u003c0xc0002cf930\u003e",
    "\u003c%!p(resolver.csKeyType=grpc.internal.resolver.configSelector)\u003e": "\u003c0xc0000beb40\u003e"
  }
} (service config updated)
2025/03/16 14:25:01 INFO: [core] [Channel #1]Channel switches to new LB policy "xds_cluster_manager_experimental"
2025/03/16 14:25:01 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Created
2025/03/16 14:25:01 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Received update from resolver, balancer config: {
  "LoadBalancingConfig": null,
  "Children": {
    "cluster:cluster-upstream.host.foo": {
      "ChildPolicy": [
        {
          "cds_experimental": {
            "LoadBalancingConfig": null,
            "Cluster": "cluster-upstream.host.foo"
          }
        }
      ]
    }
  }
}
2025/03/16 14:25:01 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Adding child policy of type "cds_experimental" for child "cluster:cluster-upstream.host.foo"
2025/03/16 14:25:01 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Creating child policy of type "cds_experimental" for child "cluster:cluster-upstream.host.foo"
2025/03/16 14:25:01 INFO: [xds] [cds-lb 0xc0003cea80] Created
2025/03/16 14:25:01 INFO: [xds] [cds-lb 0xc0003cea80] xDS credentials in use: false
2025/03/16 14:25:01 INFO: [xds] [cds-lb 0xc0003cea80] Received balancer config update: {
  "LoadBalancingConfig": null,
  "Cluster": "cluster-upstream.host.foo"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [authority ""] New watch for type "ClusterResource", resource name "cluster-upstream.host.foo"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [authority ""] First watch for type "ClusterResource", resource name "cluster-upstream.host.foo"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Subscribing to resource "cluster-upstream.host.foo" of type "ClusterResource"
2025/03/16 14:25:01 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc0003f3270,state:CONNECTING,stateToAggregate:CONNECTING]
2025/03/16 14:25:01 INFO: [core] [Channel #1]Channel Connectivity change to CONNECTING
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS request sent: {
  "resourceNames":  [
    "cluster-upstream.host.foo"
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.cluster.v3.Cluster"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS response received: {
  "versionInfo":  "1",
  "resources":  [
    {
      "@type":  "type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name":  "cluster-upstream.host.foo",
      "type":  "EDS",
      "edsClusterConfig":  {
        "edsConfig":  {
          "ads":  {},
          "resourceApiVersion":  "V3"
        }
      },
      "healthChecks":  [
        {
          "timeout":  "5s",
          "interval":  "10s"
        }
      ]
    }
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.cluster.v3.Cluster",
  "nonce":  "3"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Sending ACK for resource type: "type.googleapis.com/envoy.config.cluster.v3.Cluster", version: "1", nonce: "3"
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS request sent: {
  "versionInfo":  "1",
  "resourceNames":  [
    "cluster-upstream.host.foo"
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.cluster.v3.Cluster",
  "responseNonce":  "3"
}
2025/03/16 14:25:01 INFO: [xds] [xds-client 0xc00035a360] [authority ""] Resource type "ClusterResource" with name "cluster-upstream.host.foo" added to cache
2025/03/16 14:25:01 INFO: [xds] [cds-lb 0xc0003cea80] Received Cluster resource: {
  "ClusterType": 0,
  "ClusterName": "cluster-upstream.host.foo",
  "EDSServiceName": "",
  "LRSServerConfig": null,
  "SecurityCfg": null,
  "MaxRequests": null,
  "DNSHostName": "",
  "PrioritizedClusterNames": null,
  "LBPolicy": [
    {
      "xds_wrr_locality_experimental": {
        "childPolicy": [
          {
            "round_robin": {}
          }
        ]
      }
    }
  ],
  "OutlierDetection": null,
  "Raw": {
    "type_url": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
    "value": "ChljbHVzdGVyLXVwc3RyZWFtLmhvc3QuZm9vGgYKBDACGgBCCAoCCAUSAggKEAM="
  },
  "TelemetryLabels": {
    "csm.service_name": "unknown",
    "csm.service_namespace_name": "unknown"
  }
}
2025/03/16 14:25:02 INFO: [xds] [xds-cluster-resolver-lb 0xc0000cf1e0] Created
2025/03/16 14:25:02 INFO: [xds] [cds-lb 0xc0003cea80] Created child policy 0xc0000cf1e0 of type cluster_resolver_experimental
2025/03/16 14:25:02 INFO: [xds] [xds-cluster-resolver-lb 0xc0000cf1e0] Received new balancer config: {
  "discoveryMechanisms": [
    {
      "cluster": "cluster-upstream.host.foo",
      "outlierDetection": {},
      "telemetryLabels": {
        "csm.service_name": "unknown",
        "csm.service_namespace_name": "unknown"
      }
    }
  ],
  "xdsLbPolicy": [
    {
      "xds_wrr_locality_experimental": {
        "childPolicy": [
          {
            "round_robin": {}
          }
        ]
      }
    }
  ]
}
2025/03/16 14:25:02 INFO: [xds] [xds-client 0xc00035a360] [authority ""] New watch for type "EndpointsResource", resource name "cluster-upstream.host.foo"
2025/03/16 14:25:02 INFO: [xds] [xds-client 0xc00035a360] [authority ""] First watch for type "EndpointsResource", resource name "cluster-upstream.host.foo"
2025/03/16 14:25:02 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Subscribing to resource "cluster-upstream.host.foo" of type "EndpointsResource"
2025/03/16 14:25:02 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS request sent: {
  "resourceNames":  [
    "cluster-upstream.host.foo"
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment"
}
2025/03/16 14:25:02 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS response received: {
  "versionInfo":  "1",
  "resources":  [
    {
      "@type":  "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
      "clusterName":  "cluster-upstream.host.foo",
      "endpoints":  [
        {
          "locality":  {
            "region":  "us-regionA",
            "zone":  "az-1"
          },
          "lbEndpoints":  [
            {
              "endpoint":  {
                "address":  {
                  "socketAddress":  {
                    "address":  "127.0.0.1",
                    "portValue":  50051
                  }
                }
              },
              "healthStatus":  "HEALTHY"
            }
          ],
          "loadBalancingWeight":  44
        }
      ]
    }
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
  "nonce":  "4"
}
2025/03/16 14:25:02 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Sending ACK for resource type: "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment", version: "1", nonce: "4"
2025/03/16 14:25:02 INFO: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS request sent: {
  "versionInfo":  "1",
  "resourceNames":  [
    "cluster-upstream.host.foo"
  ],
  "typeUrl":  "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
  "responseNonce":  "4"
}
2025/03/16 14:25:02 INFO: [xds] [xds-client 0xc00035a360] [authority ""] Resource type "EndpointsResource" with name "cluster-upstream.host.foo" added to cache
2025/03/16 14:25:02 INFO: [xds] [priority-lb 0xc0001497a0] Created
2025/03/16 14:25:02 INFO: [xds] [xds-cluster-resolver-lb 0xc0000cf1e0] Built child policy config: {
  "children": {
    "priority-0-0": {
      "config": [
        {
          "outlier_detection_experimental": {
            "interval": "10s",
            "baseEjectionTime": "30s",
            "maxEjectionTime": "300s",
            "maxEjectionPercent": 10,
            "childPolicy": [
              {
                "xds_cluster_impl_experimental": {
                  "cluster": "cluster-upstream.host.foo",
                  "telemetryLabels": {
                    "csm.service_name": "unknown",
                    "csm.service_namespace_name": "unknown"
                  },
                  "childPolicy": [
                    {
                      "xds_wrr_locality_experimental": {
                        "childPolicy": [
                          {
                            "round_robin": {}
                          }
                        ]
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      ],
      "ignoreReresolutionRequests": true
    }
  },
  "priorities": [
    "priority-0-0"
  ]
}
2025/03/16 14:25:02 INFO: [xds] [priority-lb 0xc0001497a0] Received an update with balancer config: {
  "children": {
    "priority-0-0": {
      "config": [
        {
          "outlier_detection_experimental": {
            "interval": "10s",
            "baseEjectionTime": "30s",
            "maxEjectionTime": "300s",
            "maxEjectionPercent": 10,
            "childPolicy": [
              {
                "xds_cluster_impl_experimental": {
                  "cluster": "cluster-upstream.host.foo",
                  "telemetryLabels": {
                    "csm.service_name": "unknown",
                    "csm.service_namespace_name": "unknown"
                  },
                  "childPolicy": [
                    {
                      "xds_wrr_locality_experimental": {
                        "childPolicy": [
                          {
                            "round_robin": {}
                          }
                        ]
                      }
                    }
                  ]
                }
              }
            ]
          }
        }
      ],
      "ignoreReresolutionRequests": true
    }
  },
  "priorities": [
    "priority-0-0"
  ]
}
2025/03/16 14:25:02 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "", "priority-0-0"
2025/03/16 14:25:02 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:CONNECTING Picker:0xc0003a86c0}
2025/03/16 14:25:02 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:CONNECTING Picker:0xc0003a86c0}
2025/03/16 14:25:02 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc0003a86c0,state:CONNECTING,stateToAggregate:CONNECTING]
2025/03/16 14:25:02 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:25:02 INFO: [xds] [priority-lb 0xc0001497a0] Adding child policy of type "outlier_detection_experimental" for child "priority-0-0"
2025/03/16 14:25:02 INFO: [xds] [priority-lb 0xc0001497a0] Creating child policy of type "outlier_detection_experimental" for child "priority-0-0"
2025/03/16 14:25:02 INFO: [xds] [outlier-detection-lb 0xc0003aa820] Created
2025/03/16 14:25:02 INFO: [xds] [xds-cluster-impl-lb 0xc0000fb900] Created
2025/03/16 14:25:02 INFO: [xds] [xds-cluster-impl-lb 0xc0000fb900] Received configuration: {
  "cluster": "cluster-upstream.host.foo",
  "telemetryLabels": {
    "csm.service_name": "unknown",
    "csm.service_namespace_name": "unknown"
  },
  "childPolicy": [
    {
      "xds_wrr_locality_experimental": {
        "childPolicy": [
          {
            "round_robin": {}
          }
        ]
      }
    }
  ]
}
2025/03/16 14:25:02 INFO: [xds] [weighted-target-lb 0xc000349f00] Created
2025/03/16 14:25:02 INFO: [xds] [wrrlocality-lb 0xc0003a5530] Created
2025/03/16 14:25:02 INFO: [xds] [weighted-target-lb 0xc000349f00] Received update from resolver, balancer config: {
  "targets": {
    "{\"region\":\"us-regionA\",\"zone\":\"az-1\"}": {
      "weight": 44,
      "childPolicy": [
        {
          "round_robin": {}
        }
      ]
    }
  }
}
2025/03/16 14:25:02 INFO: [xds] [weighted-target-lb 0xc000349f00] Adding child policy of type "round_robin" for child "{\"region\":\"us-regionA\",\"zone\":\"az-1\"}"
2025/03/16 14:25:02 INFO: [xds] [weighted-target-lb 0xc000349f00] Creating child policy of type "round_robin" for child "{\"region\":\"us-regionA\",\"zone\":\"az-1\"}"
2025/03/16 14:25:02 INFO: [roundrobin] [0xc0003a5a10] Created
2025/03/16 14:25:02 INFO: [pick-first-leaf-lb] [pick-first-leaf-lb 0xc0003a6ea0] Received new config {
  "shuffleAddressList": false
}, resolver state {
  "Addresses": null,
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:50051",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": {
            "\u003c%!p(wrrlocality.attributeKey={})\u003e": "Locality Weight: 44",
            "\u003c%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)\u003e": "\u003c0xc000349b40\u003e",
            "\u003c%!p(internal.localityKeyType=grpc.xds.internal.address.locality)\u003e": "\u003c%!p(internal.LocalityID={us-regionA az-1 })\u003e",
            "\u003c%!p(weightedroundrobin.attributeKey={})\u003e": "Weight: 44"
          },
          "Metadata": null
        }
      ],
      "Attributes": {
        "\u003c%!p(wrrlocality.attributeKey={})\u003e": "Locality Weight: 44",
        "\u003c%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)\u003e": "\u003c0xc000349b50\u003e",
        "\u003c%!p(internal.localityKeyType=grpc.xds.internal.address.locality)\u003e": "\u003c%!p(internal.LocalityID={us-regionA az-1 })\u003e",
        "\u003c%!p(weightedroundrobin.attributeKey={})\u003e": "Weight: 44"
      }
    }
  ],
  "ServiceConfig": null,
  "Attributes": {
    "\u003c%!p(pickfirstleaf.enableHealthListenerKeyType={})\u003e": "\u003c%!p(bool=true)\u003e",
    "\u003c%!p(xdsclient.clientKeyType=grpc.xds.internal.client.Client)\u003e": "\u003c0xc0002cf930\u003e",
    "\u003c%!p(weightedtarget.localityKeyType=locality)\u003e": "{\"region\":\"us-regionA\",\"zone\":\"az-1\"}"
  }
}
2025/03/16 14:25:02 INFO: [core] [Channel #1 SubChannel #5]Subchannel created
upstream.host.foo:picker:0xc00038f698,state:READY,stateToAggregate:READY]
2025/03/16 14:25:02 INFO: [core] [Channel #1]Channel Connectivity change to READY
2025/03/16 14:25:02 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:25:42 Client: Received response- 20 message:"Hello request from client  ----> from server1"
2025/03/16 14:25:44 Client: Received response- 21 message:"Hello request from client  ----> from server1"
2025/03/16 14:25:46 Client: Received response- 22 message:"Hello request from client  ----> from server1"

Failure injection: 1. xDS 8003 goes down

2025/03/16 14:25:48 Client: Received response- 23 message:"Hello request from client  ----> from server1"
2025/03/16 14:25:49 INFO: [transport] [client-transport 0xc0000c6488] Closing: connection error: desc = "error reading from server: EOF"
2025/03/16 14:25:49 INFO: [core] [Channel #2 SubChannel #3]Subchannel Connectivity change to IDLE
2025/03/16 14:25:49 INFO: [transport] [client-transport 0xc0000c6488] loopyWriter exiting with error: connection error: desc = "error reading from server: EOF"
2025/03/16 14:25:49 WARNING: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] ADS stream closed: rpc error: code = Unavailable desc = error reading from server: EOF
2025/03/16 14:25:49 INFO: [core] blockingPicker: the picked transport is not ready, loop back to repick
2025/03/16 14:25:49 INFO: [xds] [xds-client 0xc00035a360] [authority ""] Connection to server xds.domain.com:8003-insecure-xds_v3 failed with error: rpc error: code = Unavailable desc = error reading from server: EOF
2025/03/16 14:25:49 WARNING: [xds] [xds-client 0xc00035a360] [authority ""] Watchers not notified since ADS stream failed after having received at least one response: rpc error: code = Unavailable desc = error reading from server: EOF
2025/03/16 14:25:49 INFO: [pick-first-lb] [pick-first-lb 0xc000052510] Received SubConn state update: 0xc0000be190, {ConnectivityState:IDLE ConnectionError:<nil> connectedAddress:{Addr: ServerName: Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}
2025/03/16 14:25:49 INFO: [core] [Channel #2]Channel Connectivity change to IDLE
2025/03/16 14:25:49 INFO: [core] [Channel #2 SubChannel #3]Subchannel Connectivity change to CONNECTING
2025/03/16 14:25:49 INFO: [core] [Channel #2 SubChannel #3]Subchannel picks a new address "127.0.0.1:8003" to connect
2025/03/16 14:25:49 INFO: [pick-first-lb] [pick-first-lb 0xc000052510] Received SubConn state update: 0xc0000be190, {ConnectivityState:CONNECTING ConnectionError:<nil> connectedAddress:{Addr: ServerName: Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}
2025/03/16 14:25:49 INFO: [core] [Channel #2]Channel Connectivity change to CONNECTING
2025/03/16 14:25:49 INFO: [core] Creating new client transport to "{Addr: \"127.0.0.1:8003\", ServerName: \"xds.domain.com:8003\", }": connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 WARNING: [core] [Channel #2 SubChannel #3]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:8003", ServerName: "xds.domain.com:8003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [core] [Channel #2 SubChannel #3]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [pick-first-lb] [pick-first-lb 0xc000052510] Received SubConn state update: 0xc0000be190, {ConnectivityState:TRANSIENT_FAILURE ConnectionError:connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused" connectedAddress:{Addr: ServerName: Attributes:<nil> BalancerAttributes:<nil> Metadata:<nil>}}
2025/03/16 14:25:49 INFO: [core] [Channel #2]Channel Connectivity change to TRANSIENT_FAILURE
2025/03/16 14:25:49 WARNING: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Failed to create a new ADS streaming RPC: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [xds] [xds-client 0xc00035a360] [authority ""] Connection to server xds.domain.com:8003-insecure-xds_v3 failed with error: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [xds] [xds-client 0xc00035a360] [authority ""] No watchers for uncached resources. Not triggering fallback
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-resolver-lb 0xc0000cf1e0] EDS discovery mechanism for resource "cluster-upstream.host.foo" reported error: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 WARNING: [xds] [cds-lb 0xc0003cea80] Cluster resource "cluster-upstream.host.foo" received error update: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 WARNING: [xds] [xds-cluster-resolver-lb 0xc0000cf1e0] Received error: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [pick-first-leaf-lb] [pick-first-leaf-lb 0xc0003a6ea0] Received error from the name resolver: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [pick-first-leaf-lb] [pick-first-leaf-lb 0xc0003a6ea0] Ignoring resolver error because balancer is using a previous good update.
2025/03/16 14:25:49 INFO: [xds] [weighted-target-lb 0xc000349f00] Balancer state update from child {"region":"us-regionA","zone":"az-1"}, new state: {ConnectivityState:READY Picker:0xc000559a80}
2025/03/16 14:25:49 INFO: [xds] [weighted-target-lb 0xc000349f00] Child pickers with config: map[{"region":"us-regionA","zone":"az-1"}:weight:44,picker:0xc000559a80,state:READY,stateToAggregate:READY]
2025/03/16 14:25:49 INFO: [xds] [priority-lb 0xc0001497a0] Balancer state update from child priority-0-0, new state: {ConnectivityState:READY Picker:0xc000563740}
2025/03/16 14:25:49 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:READY Picker:0xc000563740}
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:READY Picker:0xc000563740}
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc000563740,state:READY,stateToAggregate:READY]
2025/03/16 14:25:49 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:25:49 INFO: [xds] [xds-resolver 0xc000360000] Received error for RouteConfiguration resource "route-upstream.host.foo": [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 WARNING: [core] [Channel #1]ccResolverWrapper: reporting error to cc: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [xds] [xds-resolver 0xc000360000] Received error for Listener resource "upstream.host.foo": [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 WARNING: [core] [Channel #1]ccResolverWrapper: reporting error to cc: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 WARNING: [xds] [cds-lb 0xc0003cea80] Cluster resource "cluster-upstream.host.foo" received error update: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 WARNING: [xds] [cds-lb 0xc0003cea80] Cluster resource "cluster-upstream.host.foo" received error update: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 WARNING: [xds] [xds-cluster-resolver-lb 0xc0000cf1e0] Received error: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [pick-first-leaf-lb] [pick-first-leaf-lb 0xc0003a6ea0] Received error from the name resolver: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [pick-first-leaf-lb] [pick-first-leaf-lb 0xc0003a6ea0] Ignoring resolver error because balancer is using a previous good update.
2025/03/16 14:25:49 INFO: [xds] [weighted-target-lb 0xc000349f00] Balancer state update from child {"region":"us-regionA","zone":"az-1"}, new state: {ConnectivityState:READY Picker:0xc000559ac0}
2025/03/16 14:25:49 INFO: [xds] [weighted-target-lb 0xc000349f00] Child pickers with config: map[{"region":"us-regionA","zone":"az-1"}:weight:44,picker:0xc000559ac0,state:READY,stateToAggregate:READY]
2025/03/16 14:25:49 WARNING: [xds] [xds-cluster-resolver-lb 0xc0000cf1e0] Received error: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [pick-first-leaf-lb] [pick-first-leaf-lb 0xc0003a6ea0] Received error from the name resolver: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:25:49 INFO: [pick-first-leaf-lb] [pick-first-leaf-lb 0xc0003a6ea0] Ignoring resolver error because balancer is using a previous good update.
2025/03/16 14:25:49 INFO: [xds] [weighted-target-lb 0xc000349f00] Balancer state update from child {"region":"us-regionA","zone":"az-1"}, new state: {ConnectivityState:READY Picker:0xc000559b00}
2025/03/16 14:25:49 INFO: [xds] [weighted-target-lb 0xc000349f00] Child pickers with config: map[{"region":"us-regionA","zone":"az-1"}:weight:44,picker:0xc000559b00,state:READY,stateToAggregate:READY]
2025/03/16 14:25:49 INFO: [xds] [priority-lb 0xc0001497a0] Balancer state update from child priority-0-0, new state: {ConnectivityState:READY Picker:0xc000563818}
2025/03/16 14:25:49 INFO: [xds] [priority-lb 0xc0001497a0] Balancer state update from child priority-0-0, new state: {ConnectivityState:READY Picker:0xc000563848}
2025/03/16 14:25:49 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:READY Picker:0xc000563818}
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:READY Picker:0xc000563818}
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc000563818,state:READY,stateToAggregate:READY]
2025/03/16 14:25:49 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:25:49 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:READY Picker:0xc000563848}
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:READY Picker:0xc000563848}
2025/03/16 14:25:49 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc000563848,state:READY,stateToAggregate:READY]
2025/03/16 14:25:49 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:25:50 INFO: [core] [Channel #2]Resolver state updated: {
  "Addresses": [
    {
      "Addr": "127.0.0.1:8003",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:8003",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} ()
2025/03/16 14:25:50 INFO: [pick-first-lb] [pick-first-lb 0xc000052510] Received new config {
  "shuffleAddressList": false
}, resolver state {
  "Addresses": [
    {
      "Addr": "127.0.0.1:8003",
      "ServerName": "",
      "Attributes": null,
      "BalancerAttributes": null,
      "Metadata": null
    }
  ],
  "Endpoints": [
    {
      "Addresses": [
        {
          "Addr": "127.0.0.1:8003",
          "ServerName": "",
          "Attributes": null,
          "BalancerAttributes": null,
          "Metadata": null
        }
      ],
      "Attributes": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
}
2025/03/16 14:25:50 INFO: [core] [Channel #2 SubChannel #3]addrConn: updateAddrs addrs (1 of 1): [{Addr: "127.0.0.1:8003", ServerName: "", }]
2025/03/16 14:25:50 Client: Received response- 24 message:"Hello request from client  ----> from server1"
^Csignal: interrupt
``
Failure injection: 2. gRPC server1 goes down: 

<!-- Failed to upload "Failover_not_working.txt" -->

2025/03/16 14:26:00 Client: Received response- 29 message:"Hello request from client  ----> from server1"
2025/03/16 14:26:00 INFO: [transport] [client-transport 0xc0003ac248] Closing: connection error: desc = "error reading from server: EOF"
2025/03/16 14:26:00 INFO: [core] [Channel #1 SubChannel #5]Subchannel Connectivity change to IDLE
2025/03/16 14:26:00 INFO: [transport] [client-transport 0xc0003ac248] loopyWriter exiting with error: connection error: desc = "error reading from server: EOF"
2025/03/16 14:26:00 INFO: [xds] [weighted-target-lb 0xc000349f00] Balancer state update from child {"region":"us-regionA","zone":"az-1"}, new state: {ConnectivityState:IDLE Picker:0xc000466040}
2025/03/16 14:26:00 INFO: [xds] [weighted-target-lb 0xc000349f00] Child pickers with config: map[{"region":"us-regionA","zone":"az-1"}:weight:44,picker:0xc000466040,state:IDLE,stateToAggregate:IDLE]
2025/03/16 14:26:00 INFO: [xds] [priority-lb 0xc0001497a0] Balancer state update from child priority-0-0, new state: {ConnectivityState:IDLE Picker:0xc0005ac078}
2025/03/16 14:26:00 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/03/16 14:26:00 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:IDLE Picker:0xc0005ac078}
2025/03/16 14:26:00 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:IDLE Picker:0xc0005ac078}
2025/03/16 14:26:00 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc0005ac078,state:IDLE,stateToAggregate:IDLE]
2025/03/16 14:26:00 INFO: [core] [Channel #1]Channel Connectivity change to IDLE
2025/03/16 14:26:00 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:26:00 INFO: [core] [Channel #1 SubChannel #5]Subchannel Connectivity change to CONNECTING
2025/03/16 14:26:00 INFO: [core] [Channel #1 SubChannel #5]Subchannel picks a new address "127.0.0.1:50051" to connect
2025/03/16 14:26:00 INFO: [xds] [weighted-target-lb 0xc000349f00] Balancer state update from child {"region":"us-regionA","zone":"az-1"}, new state: {ConnectivityState:CONNECTING Picker:0xc000466100}
2025/03/16 14:26:00 INFO: [xds] [weighted-target-lb 0xc000349f00] Child pickers with config: map[{"region":"us-regionA","zone":"az-1"}:weight:44,picker:0xc000466100,state:CONNECTING,stateToAggregate:CONNECTING]
2025/03/16 14:26:00 INFO: [xds] [priority-lb 0xc0001497a0] Balancer state update from child priority-0-0, new state: {ConnectivityState:CONNECTING Picker:0xc0005ac120}
2025/03/16 14:26:00 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/03/16 14:26:00 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:CONNECTING Picker:0xc0005ac120}
2025/03/16 14:26:00 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:CONNECTING Picker:0xc0005ac120}
2025/03/16 14:26:00 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc0005ac120,state:CONNECTING,stateToAggregate:CONNECTING]
2025/03/16 14:26:00 INFO: [core] [Channel #1]Channel Connectivity change to CONNECTING
2025/03/16 14:26:00 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:26:00 INFO: [core] Creating new client transport to "{Addr: \"127.0.0.1:50051\", ServerName: \"upstream.host.foo\", Attributes: {\"<%!p(xds.handshakeClusterNameKey={})>\": \"cluster-upstream.host.foo\" , \"<%!p(xds.handshakeAttrKey={})>\": \"<0xc00007eee0>\" }, BalancerAttributes: {\"<%!p(wrrlocality.attributeKey={})>\": \"Locality Weight: 44\" , \"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>\": \"<0xc000349b40>\" , \"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>\": \"<%!p(internal.LocalityID={us-regionA az-1 })>\" , \"<%!p(weightedroundrobin.attributeKey={})>\": \"Weight: 44\" , \"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" }}": connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:00 WARNING: [core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:50051", ServerName: "upstream.host.foo", Attributes: {"<%!p(xds.handshakeClusterNameKey={})>": "cluster-upstream.host.foo" , "<%!p(xds.handshakeAttrKey={})>": "<0xc00007eee0>" }, BalancerAttributes: {"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>": "<%!p(internal.LocalityID={us-regionA az-1 })>" , "<%!p(weightedroundrobin.attributeKey={})>": "Weight: 44" , "<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" , "<%!p(wrrlocality.attributeKey={})>": "Locality Weight: 44" , "<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>": "<0xc000349b40>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:00 INFO: [core] [Channel #1 SubChannel #5]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:00 INFO: [xds] [weighted-target-lb 0xc000349f00] Balancer state update from child {"region":"us-regionA","zone":"az-1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004661c0}
2025/03/16 14:26:00 INFO: [xds] [weighted-target-lb 0xc000349f00] Child pickers with config: map[{"region":"us-regionA","zone":"az-1"}:weight:44,picker:0xc0004661c0,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/03/16 14:26:00 INFO: [xds] [priority-lb 0xc0001497a0] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0005ac210}
2025/03/16 14:26:00 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/03/16 14:26:00 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0005ac210}
2025/03/16 14:26:00 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0005ac210}
2025/03/16 14:26:00 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc0005ac210,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/03/16 14:26:00 INFO: [core] [Channel #1]Channel Connectivity change to TRANSIENT_FAILURE
2025/03/16 14:26:00 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:26:01 INFO: [core] [Channel #1 SubChannel #5]Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:01 INFO: [core] [Channel #1 SubChannel #5]Subchannel Connectivity change to CONNECTING
2025/03/16 14:26:01 INFO: [core] [Channel #1 SubChannel #5]Subchannel picks a new address "127.0.0.1:50051" to connect
2025/03/16 14:26:01 INFO: [core] Creating new client transport to "{Addr: \"127.0.0.1:50051\", ServerName: \"upstream.host.foo\", Attributes: {\"<%!p(xds.handshakeClusterNameKey={})>\": \"cluster-upstream.host.foo\" , \"<%!p(xds.handshakeAttrKey={})>\": \"<0xc00007eee0>\" }, BalancerAttributes: {\"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" , \"<%!p(wrrlocality.attributeKey={})>\": \"Locality Weight: 44\" , \"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>\": \"<0xc000349b40>\" , \"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>\": \"<%!p(internal.LocalityID={us-regionA az-1 })>\" , \"<%!p(weightedroundrobin.attributeKey={})>\": \"Weight: 44\" }}": connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:01 WARNING: [core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:50051", ServerName: "upstream.host.foo", Attributes: {"<%!p(xds.handshakeClusterNameKey={})>": "cluster-upstream.host.foo" , "<%!p(xds.handshakeAttrKey={})>": "<0xc00007eee0>" }, BalancerAttributes: {"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>": "<%!p(internal.LocalityID={us-regionA az-1 })>" , "<%!p(weightedroundrobin.attributeKey={})>": "Weight: 44" , "<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" , "<%!p(wrrlocality.attributeKey={})>": "Locality Weight: 44" , "<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>": "<0xc000349b40>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:01 INFO: [core] [Channel #1 SubChannel #5]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:01 INFO: [xds] [weighted-target-lb 0xc000349f00] Balancer state update from child {"region":"us-regionA","zone":"az-1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc000466280}
2025/03/16 14:26:01 INFO: [xds] [weighted-target-lb 0xc000349f00] Child pickers with config: map[{"region":"us-regionA","zone":"az-1"}:weight:44,picker:0xc000466280,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/03/16 14:26:01 INFO: [xds] [priority-lb 0xc0001497a0] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0005ac2e8}
2025/03/16 14:26:01 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/03/16 14:26:01 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0005ac2e8}
2025/03/16 14:26:01 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0005ac2e8}
2025/03/16 14:26:01 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc0005ac2e8,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/03/16 14:26:01 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:26:03 INFO: [core] [Channel #1 SubChannel #5]Subchannel Connectivity change to IDLE, last error: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:03 INFO: [core] [Channel #1 SubChannel #5]Subchannel Connectivity change to CONNECTING
2025/03/16 14:26:03 INFO: [core] [Channel #1 SubChannel #5]Subchannel picks a new address "127.0.0.1:50051" to connect
2025/03/16 14:26:03 INFO: [core] Creating new client transport to "{Addr: \"127.0.0.1:50051\", ServerName: \"upstream.host.foo\", Attributes: {\"<%!p(xds.handshakeClusterNameKey={})>\": \"cluster-upstream.host.foo\" , \"<%!p(xds.handshakeAttrKey={})>\": \"<0xc00007eee0>\" }, BalancerAttributes: {\"<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>\": \"<%!p(internal.LocalityID={us-regionA az-1 })>\" , \"<%!p(weightedroundrobin.attributeKey={})>\": \"Weight: 44\" , \"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>\": \"<%!p(bool=true)>\" , \"<%!p(wrrlocality.attributeKey={})>\": \"Locality Weight: 44\" , \"<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>\": \"<0xc000349b40>\" }}": connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:03 WARNING: [core] [Channel #1 SubChannel #5]grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:50051", ServerName: "upstream.host.foo", Attributes: {"<%!p(xds.handshakeClusterNameKey={})>": "cluster-upstream.host.foo" , "<%!p(xds.handshakeAttrKey={})>": "<0xc00007eee0>" }, BalancerAttributes: {"<%!p(wrrlocality.attributeKey={})>": "Locality Weight: 44" , "<%!p(hierarchy.pathKeyType=grpc.internal.address.hierarchical_path)>": "<0xc000349b40>" , "<%!p(internal.localityKeyType=grpc.xds.internal.address.locality)>": "<%!p(internal.LocalityID={us-regionA az-1 })>" , "<%!p(weightedroundrobin.attributeKey={})>": "Weight: 44" , "<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:03 INFO: [core] [Channel #1 SubChannel #5]Subchannel Connectivity change to TRANSIENT_FAILURE, last error: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:50051: connect: connection refused"
2025/03/16 14:26:03 INFO: [xds] [weighted-target-lb 0xc000349f00] Balancer state update from child {"region":"us-regionA","zone":"az-1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0004663c0}
2025/03/16 14:26:03 INFO: [xds] [weighted-target-lb 0xc000349f00] Child pickers with config: map[{"region":"us-regionA","zone":"az-1"}:weight:44,picker:0xc0004663c0,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/03/16 14:26:03 INFO: [xds] [priority-lb 0xc0001497a0] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0005ac3c0}
2025/03/16 14:26:03 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/03/16 14:26:03 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0005ac3c0}
2025/03/16 14:26:03 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0005ac3c0}
2025/03/16 14:26:03 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc0005ac3c0,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/03/16 14:26:03 INFO: [xds] [priority-lb 0xc0001497a0] Switching to ("priority-0-0", 0) in syncPriority
2025/03/16 14:26:04 WARNING: [xds] [xds-client 0xc00035a360] [xds-channel 0xc00036e120] [ads-stream 0xc000296f80] Failed to create a new ADS streaming RPC: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:26:04 INFO: [xds] [xds-client 0xc00035a360] [authority ""] Connection to server xds.domain.com:8003-insecure-xds_v3 failed with error: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:26:04 INFO: [xds] [xds-client 0xc00035a360] [authority ""] No watchers for uncached resources. Not triggering fallback
2025/03/16 14:26:04 INFO: [xds] [xds-cluster-resolver-lb 0xc0000cf1e0] EDS discovery mechanism for resource "cluster-upstream.host.foo" reported error: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:26:04 WARNING: [xds] [cds-lb 0xc0003cea80] Cluster resource "cluster-upstream.host.foo" received error update: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:26:04 WARNING: [xds] [xds-cluster-resolver-lb 0xc0000cf1e0] Received error: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:26:04 INFO: [pick-first-leaf-lb] [pick-first-leaf-lb 0xc0003a6ea0] Received error from the name resolver: [xDS node id: test-foo]: xds: error received from xDS stream: failed to create an ADS stream: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:8003: connect: connection refused"
2025/03/16 14:26:04 INFO: [xds] [weighted-target-lb 0xc000349f00] Balancer state update from child {"region":"us-regionA","zone":"az-1"}, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc00062e200}
2025/03/16 14:26:04 INFO: [xds] [weighted-target-lb 0xc000349f00] Child pickers with config: map[{"region":"us-regionA","zone":"az-1"}:weight:44,picker:0xc00062e200,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]
2025/03/16 14:26:04 INFO: [xds] [priority-lb 0xc0001497a0] Balancer state update from child priority-0-0, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0006321b0}
2025/03/16 14:26:04 INFO: [xds] [priority-lb 0xc0001497a0] childInUse, childUpdating: "priority-0-0", "priority-0-0"
2025/03/16 14:26:04 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Balancer state update from child cluster:cluster-upstream.host.foo, new state: {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0006321b0}
2025/03/16 14:26:04 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] State update from sub-balancer "cluster:cluster-upstream.host.foo": {ConnectivityState:TRANSIENT_FAILURE Picker:0xc0006321b0}
2025/03/16 14:26:04 INFO: [xds] [xds-cluster-manager-lb 0xc0000a5de0] Child pickers: map[cluster:cluster-upstream.host.foo:picker:0xc0006321b0,state:TRANSIENT_FAILURE,stateToAggregate:TRANSIENT_FAILURE]

Software in use:

go 1.23.0
google.golang.org/grpc v1.71.0

Please let me know, if I missed anything.

[purnesh42H]

@axkum10 thanks for providing the debug logs. I see in your logs No watchers for uncached resources. Not triggering fallback which means there is no watch happening for uncached resource. It is because network fault happened after receiving atleast 1 good update from management server. See log line Watchers not notified since ADS stream failed after having received at least one response.

Two conditions need to be met for fallback to be triggered:

1. There is a connectivity failure on the ADS stream, as described in gRFC A57. For us, this means that the ADS stream was closed before the first server response was received.
2. There is at least one watcher for a resource that is not cached. Cached resources include ones that
   • have been successfully received and can be used.
   • are considered non-existent according to xDS Protocol Specification.

Edit: Linking A71 https://github.com/grpc/proposal/blob/master/A71-xds-fallback.md#reservations-about-using-the-fallback-server-data for the background on why we always prefer a cached update from primary server over fallback server for the same resource.

[github-actions]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

[github-actions]

This issue is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.