Default retry logic does not retry HTTP 503 errors from BigQuery API on `BigQuery.getTable()`

[patrykks]

[bigquery]: Default retry logic does not retry HTTP 503 errors from BigQuery API on BigQuery.getTable()

When a BigQuery RPC such as BigQuery.getTable(...) receives a 503 backendError response, the call fails immediately on the first attempt. The configured RetrySettings (maxAttempts, backoff, etc.) are not honored, even though 503 is declared as retryable by the SDK itself in BigQueryException.RETRYABLE_ERRORS.

Environment details

1. API: BigQuery
2. Java version: 21
3. Version(s):
   • com.google.cloud:google-cloud-bigquery:2.62.0
   • com.google.http-client:google-http-client-test:1.44.1 (used only by the reproduction below to mock the transport)

Steps to reproduce

1. Build and run the code under "Code example" with com.google.cloud:google-cloud-bigquery:2.62.0.
2. The mock transport returns HTTP 503 on the first request and HTTP 200 on the second. With RetrySettings.setMaxAttempts(5) configured, two requests are expected (first is retried, second succeeds).
3. Observe that only one request is sent and a BigQueryException is thrown immediately - the retry never happens.

Code example

package com.example;

import com.google.api.client.testing.http.MockHttpTransport;
import com.google.api.client.testing.http.MockLowLevelHttpRequest;
import com.google.api.client.testing.http.MockLowLevelHttpResponse;
import com.google.api.gax.retrying.RetrySettings;
import com.google.auth.oauth2.AccessToken;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.cloud.bigquery.BigQuery;
import com.google.cloud.bigquery.BigQueryOptions;
import com.google.cloud.bigquery.Table;
import com.google.cloud.bigquery.TableId;
import com.google.cloud.http.HttpTransportOptions;
import org.threeten.bp.Duration;

import java.util.Date;

public class Repro {

    public static void main(String[] args) {
        MockHttpTransport mockTransport = new MockHttpTransport() {
            private int requestCount = 0;

            @Override
            public MockLowLevelHttpRequest buildRequest(String method, String url) {
                return new MockLowLevelHttpRequest() {
                    @Override
                    public MockLowLevelHttpResponse execute() {
                        requestCount++;
                        System.out.println("[Mock] Intercepted request #" + requestCount + " to: " + url);

                        if (requestCount < 2) {
                            String jsonError = """
                                    {
                                      "error": {
                                        "code": 503,
                                        "message": "Visibility check was unavailable. Please retry the request",
                                        "errors": [{"reason": "backendError", "domain": "global"}]
                                      }
                                    }
                                    """;
                            return new MockLowLevelHttpResponse()
                                    .setStatusCode(503)
                                    .setContentType("application/json")
                                    .setContent(jsonError);
                        }

                        String tableJson = """
                                {
                                  "kind": "bigquery#table",
                                  "id": "test-project:test_dataset.test_table",
                                  "type": "TABLE",
                                  "tableReference": {
                                    "projectId": "test-project",
                                    "datasetId": "test_dataset",
                                    "tableId": "test_table"
                                  }
                                }
                                """;
                        return new MockLowLevelHttpResponse()
                                .setStatusCode(200)
                                .setContentType("application/json")
                                .setContent(tableJson);
                    }
                };
            }
        };

        BigQueryOptions options = BigQueryOptions.newBuilder()
                .setProjectId("test-project")
                .setCredentials(GoogleCredentials.create(
                        new AccessToken("dummy-token", new Date(System.currentTimeMillis() + 3600000))))
                .setRetrySettings(RetrySettings.newBuilder()
                        .setMaxAttempts(5)
                        .setInitialRetryDelay(Duration.ofSeconds(1))
                        .setRetryDelayMultiplier(2.0)
                        .setMaxRetryDelay(Duration.ofSeconds(10))
                        .setTotalTimeout(Duration.ofSeconds(30))
                        .build())
                .setTransportOptions(HttpTransportOptions.newBuilder()
                        .setHttpTransportFactory(() -> mockTransport)
                        .build())
                .build();

        BigQuery bigquery = options.getService();

        Table table = bigquery.getTable(TableId.of("test_dataset", "test_table"));
        System.out.println("Got table: " + table.getTableId().getTable());
    }
}

Stack trace

[Mock] Intercepted request #1 to: https://bigquery.googleapis.com/...
Exception in thread "main" com.google.cloud.bigquery.BigQueryException: Visibility check was unavailable. Please retry the request
    at com.google.cloud.bigquery.BigQueryRetryHelper.runWithRetries(...)
    ...

Only one request is made. Expected: two requests, then a successful Table.

External references such as API reference guides

• com.google.cloud.bigquery.BigQueryException.RETRYABLE_ERRORS in googleapis/java-bigquery - lists {500, 502, 503, 504} as retryable.
• BigQueryOptions.Builder.setResultRetryAlgorithm(...) - the public hook that currently has to be used to make 5xx retries actually happen.

Any additional information below

Expected behavior. A 503 backendError response is retried according to the configured RetrySettings, because:

• BigQueryException.RETRYABLE_ERRORS explicitly lists {500, 502, 503, 504} as retryable.
• BigQuery's public documentation describes 5xx as transient and retryable.
• A user configuring RetrySettings.setMaxAttempts(5) reasonably expects 5xx to be included.

Actual behavior. The request fails on the first attempt. Total elapsed time is a single RPC round-trip - no backoff, no retry.

Why this matters / how it was discovered. The issue surfaced during a production incidents: the BigQuery API (GET https://bigquery.googleapis.com/bigquery/v2/projects/<EXMAPLE_PROJECT>/datasets/<EXAMPLE_DATASET>>/tables/<EXAMPLE_TABLE>?prettyPrint=false&view=STORAGE_STATS) sporadically returned

{
  "error": {
    "code": 503,
    "message": "Visibility check was unavailable. Please retry the request",
    "errors": [{"reason": "backendError", "domain": "global"}]
  }
}

in response to bigquery.getTable(...) calls. The default retry policy in google-cloud-bigquery did not retry these errors, and there is no straightforward configuration-level way for users to enable retrying of 5xx responses without replacing the retry algorithm wholesale via setResultRetryAlgorithm(...).

Happy to provide any additional context if useful.