[CI] GCSRepositoryAnalysisRestIT testRepositoryAnalysis failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #11472 / third-party_gcs
• elasticsearch-periodic #11442 / third-party_gcs

Reproduction Line:

./gradlew ":x-pack:plugin:snapshot-repo-test-kit:qa:gcs:javaRestTest" --tests "org.elasticsearch.repositories.blobstore.testkit.analyze.GCSRepositoryAnalysisRestIT.testRepositoryAnalysis" -Dtests.seed=A12F2A26BE32E066 -Dtests.locale=hu-HU -Dtests.timezone=America/Santarem -Druntime.java=25

Applicable branches:
9.2

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:36569], URI [/_snapshot/repository/_analyze?blob_count=10&seed=1723388546915399184&max_blob_size=1mb&timeout=120s&concurrency=4], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. See [https://www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-snapshot-repository-analyze?version=9.2] for further information. Elasticsearch attempted to remove the data it wrote at [temp-analysis-0PmUjLGtTn6Y_XzmYBFuwQ] but may have left some behind. If so, please now remove this data manually."}],"type":"repository_verification_exception","reason":"[repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it
[truncated]

Issue Reasons:

• [9.2] 2 failures in test testRepositoryAnalysis (1.1% fail rate in 174 executions)
• [9.2] 2 failures in step third-party_gcs (33.3% fail rate in 6 executions)
• [9.2] 2 failures in pipeline elasticsearch-periodic (33.3% fail rate in 6 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[joshua-adams-1]

The exception is:

[2025-11-20T10:31:54,802][INFO ][o.e.r.RepositoriesService] [test-cluster-0] put repository [default/repository] |  
-- | --
[2025-11-20T10:31:55,167][INFO ][o.e.r.b.t.a.RepositoryAnalyzeAction] [test-cluster-0] running analysis of repository [repository] using path [temp-analysis-0PmUjLGtTn6Y_XzmYBFuwQ] |  
[2025-11-20T10:32:01,653][WARN ][r.suppressed             ] [test-cluster-0] path: /_snapshot/repository/_analyze, params: {seed=1723388546915399184, max_blob_size=1mb, blob_count=10, repository=repository, timeout=120s, concurrency=4}, status: 500 org.elasticsearch.repositories.RepositoryVerificationException: [repository] Elasticsearch observed the storage system underneath this repository behaved incorrectly which indicates it is not suitable for use with Elasticsearch snapshots. See [https://www.elastic.co/docs/api/doc/elasticsearch/v9/operation/operation-snapshot-repository-analyze?version=9.2] for further information. Elasticsearch attempted to remove the data it wrote at [temp-analysis-0PmUjLGtTn6Y_XzmYBFuwQ] but may have left some behind. If so, please now remove this data manually. |  
Caused by: java.io.IOException: Exception when deleting blobs [BlobId{bucket=elasticsearch-ci-thirdparty, name=9.2_repository_test_kit_testsA12F2A26BE32E066/temp-analysis-0PmUjLGtTn6Y_XzmYBFuwQ/test-register-uncontended-LJ8yykk5REquk9O3oPox1g, generation=null}] |  
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.deleteBlobs(GoogleCloudStorageBlobStore.java:611) |  
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.deleteDirectory(GoogleCloudStorageBlobStore.java:530) |  
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobContainer.delete(GoogleCloudStorageBlobContainer.java:117) |  
at org.elasticsearch.repositories.blobstore.testkit@9.2.2-SNAPSHOT/org.elasticsearch.repositories.blobstore.testkit.analyze.RepositoryAnalyzeAction$AsyncAction.deleteContainer(RepositoryAnalyzeAction.java:881) |  
at org.elasticsearch.repositories.blobstore.testkit@9.2.2-SNAPSHOT/org.elasticsearch.repositories.blobstore.testkit.analyze.RepositoryAnalyzeAction$AsyncAction.lambda$runCleanUp$8(RepositoryAnalyzeAction.java:835) |  
at org.elasticsearch.server@9.2.2-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101) |  
at org.elasticsearch.server@9.2.2-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1088) |  
at org.elasticsearch.server@9.2.2-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27) |  
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) |  
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) |  
at java.base/java.lang.Thread.run(Thread.java:1474) |  
Caused by: com.google.cloud.storage.StorageException: The object elasticsearch-ci-thirdparty/9.2_repository_test_kit_testsA12F2A26BE32E066/temp-analysis-0PmUjLGtTn6Y_XzmYBFuwQ/test-register-uncontended-LJ8yykk5REquk9O3oPox1g exceeded the rate limit for object mutation operations (create, update, and delete). Please reduce your request rate. See https://cloud.google.com/storage/docs/gcs429. |  
at com.google.cloud.storage.StorageBatch$1.onFailure(StorageBatch.java:171) |  
at com.google.cloud.storage.spi.v1.HttpStorageRpc$1.onFailure(HttpStorageRpc.java:354) |  
at com.google.api.client.googleapis.batch.json.JsonBatchCallback.onFailure(JsonBatchCallback.java:50) |  
at com.google.api.client.googleapis.batch.json.JsonBatchCallback.onFailure(JsonBatchCallback.java:46) |  
at com.google.api.client.googleapis.batch.BatchUnparsedResponse.parseAndCallback(BatchUnparsedResponse.java:209) |  
at com.google.api.client.googleapis.batch.BatchUnparsedResponse.parseNextResponse(BatchUnparsedResponse.java:149) |  
at com.google.api.client.googleapis.batch.BatchRequest.execute(BatchRequest.java:269) |  
at com.google.cloud.storage.spi.v1.HttpStorageRpc$DefaultRpcBatch.submit(HttpStorageRpc.java:326) |  
at com.google.cloud.storage.StorageBatch.submit(StorageBatch.java:159) |  
at org.elasticsearch.repositories.gcs.GoogleCloudStorageBlobStore.deleteBlobs(GoogleCloudStorageBlobStore.java:603) |  
... 10 more

[joshua-adams-1]

I can think of a couple of ways forward:

1. Make ES more resilient to 429 errors, implementing an exponential back off and retry policy.
2. We could try renaming the bucket to be unique to each test (if possible). This would ensure that multiple tests running at the same time, accessing the same bucket, would not cause ES to be rate limited.

I would like to know why this started recently:

[joshua-adams-1]

According to the GCS docs:

Cloud Storage is a multi-tenant service, meaning that users share the same set of underlying resources. 
In order to make the best use of these shared resources, buckets have an initial IO capacity of:

Approximately 1000 object write requests per second, which includes uploading, updating, and deleting objects. 
Note that Cloud Storage also has a smaller limitfor repeated writes to the same object name.

Approximately 5000 object read requests per second, which includes listing objects, reading object data, 
and reading object metadata.

These initial read and write rates average to 2.5PB written and 13PB read in a month for 1MB objects. 
As the request rate for a given bucket grows, Cloud Storage autoscales and automatically increases the 
IO capacity for that bucket by distributing the request load across multiple servers.

The bucket creation / deletion rate is 1 req / 2 sec (here).

The maximum rate of writes to the same object name is one write per second. Writing to the same object name at a rate above the limit might result in throttling errors. (here)

Further steps:

1. Are we creating the bucket in each test and then deleting it?
2. If there is throttling at 1000 requests a second, are we exceeding that?
3. Does creating and then deleting an object within a second count as 2 writes? The line from the docs Approximately 1000 object write requests per second, which includes uploading, updating, and deleting objects. implies to me that it does

For 2, this is the request made:

final Request request = new Request(HttpPost.METHOD_NAME, "/_snapshot/" + repository + "/_analyze");
        request.addParameter("blob_count", "10");
        request.addParameter("concurrency", "4");
        request.addParameter("max_blob_size", randomFrom("1mb", "10mb"));
        request.addParameter("timeout", timeout.getStringRep());
        request.addParameter("seed", Long.toString(randomLong()));
        request.setOptions(
            RequestOptions.DEFAULT.toBuilder()
                .setRequestConfig(RequestConfig.custom().setSocketTimeout(Math.toIntExact(timeout.millis() + 10_000)).build())
        );

A max of ten blobs should not exceed this limit.

[joshua-adams-1]

Are we creating the bucket in each test and then deleting it?

RepositoryAnalyzeAction uses the same bucket but a different path, here, so I think we can rule this out as a potential

[DaveCTurner]

Make ES more resilient to 429 errors, implementing an exponential back off and retry policy.

I'd be inclined to do that. Indeed, I would expect the SDK already does this on our behalf to some extent but maybe we just need to configure it to be more tenacious.

[mhl-b]

Make ES more resilient to 429 errors, implementing an exponential back off and retry policy.

We use defaults:

RetrySettings{
totalTimeoutDuration=PT50S, 
initialRetryDelayDuration=PT1S, 
retryDelayMultiplier=2.0, 
maxRetryDelayDuration=PT32S, 
maxAttempts=6, 
jittered=true, 
initialRpcTimeoutDuration=PT50S, 
rpcTimeoutMultiplier=1.0, 
maxRpcTimeoutDuration=PT50S
}

There are 5 retries within 30-50 seconds. totalTimeoutDuration is an absolute limit starting from attempt#0. initialRetryDelayDuration is wait time for attempt#1. retryDelayMultiplier grows exponentially initialRetryDelayDuration for consecutive retries. maxAttempts is another absolute limit, retries are done when totalTimeoutDuration or maxAttempts limit is reached.

Assuming each attempt is instantaneous then retries happen on: [0s, 1s, 3s, 7s, 15s, 31s, 63s(already timeout at 50s)] with some jitter.

I think defaults are reasonable, 5 attempts spread within 50 seconds. We can increase retries further, but I'm wondering why GCP throttled our test for a long time, there might be other tests running in the same account, taking up bandwidth.

Reducing risk to low.