test: fix flaky xds_failover StartupPrimaryNotResponding for GoogleGrpc TRANSIENT_FAILURE

[Copilot]

XdsFailoverAdsIntegrationTest.StartupPrimaryNotResponding/15 (IPv6, GoogleGrpc, UnifiedDelta) fails intermittently under ASAN due to a race between gRPC's real-time channel backoff and Envoy's simulated-time retry timer.

Root cause

After the first primaryConnectionFailure() closes the TCP connection, the gRPC channel enters TRANSIENT_FAILURE with a ~1s real-time backoff. waitForPrimaryXdsRetryTimer() advances simulated time, firing Envoy's retry timer before that real-time backoff expires. The gRPC library immediately rejects the establishNewStream() attempt (no TCP connection made), which counts as the 2nd consecutive failure and triggers failover. The test's second primaryConnectionFailure() then waits 30s (ASAN DefaultTimeout) for a primary TCP connection that never arrives — Envoy has already switched to failover — hitting the RELEASE_ASSERT.

This race is timing-sensitive: for IPv4 the gRPC library fast-retries before onEstablishmentFailure fires (channel stays in CONNECTING), so the race doesn't manifest there.

Fix

In StartupPrimaryNotResponding, before the second primaryConnectionFailure() call, poll (real-time only, no simulated clock advancement) for cds.update_failure ≥ 2 within 200×TIMEOUT_FACTOR ms. If the counter reaches 2, the 2nd failure already occurred via TRANSIENT_FAILURE and the explicit TCP-close step is skipped:

bool second_failure_via_transient_failure = false;
if (clientType() == Grpc::ClientType::GoogleGrpc) {
  const auto deadline = absl::Now() + absl::Milliseconds(200 * TIMEOUT_FACTOR);
  constexpr absl::Duration kPollInterval = absl::Milliseconds(5);
  while (absl::Now() < deadline) {
    const auto counter = TestUtility::findCounter(
        test_server_->statStore(), "cluster_manager.cds.update_failure");
    if (counter != nullptr && counter->value() >= 2) {
      second_failure_via_transient_failure = true;
      break;
    }
    absl::SleepFor(kPollInterval);
  }
}
if (!second_failure_via_transient_failure) {
  primaryConnectionFailure();
  ASSERT_TRUE(xds_connection_->waitForDisconnect());
}

Polling uses real time (not advanceTimeWait) to avoid firing additional simulated timers that could interfere with test state. The 200×TIMEOUT_FACTOR window is well above the typical <60ms propagation latency for the gRPC completion queue → Envoy main thread path.

Files changed

• test/extensions/config_subscription/grpc/xds_failover_integration_test.cc — TRANSIENT_FAILURE detection + absl/time includes
• test/extensions/config_subscription/grpc/BUILD — added @abseil-cpp//absl/time dep

[repokitteh-read-only]

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #46006 was opened by Copilot.

see: more, trace.