[CI] S3BlobStoreRepositoryTests testFailIfAlreadyExists failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #28390 / part1
• elasticsearch-pull-request #91767 / part-1
• elasticsearch-pull-request #91750 / part-1
• elasticsearch-intake #28355 / part1

Reproduction Line:

./gradlew ":modules:repository-s3:internalClusterTest" --tests "org.elasticsearch.repositories.s3.S3BlobStoreRepositoryTests.testFailIfAlreadyExists" -Dtests.seed=87A06787B36CF7DD -Dtests.locale=en-CA -Dtests.timezone=MIT -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: null

Issue Reasons:

• [main] 4 failures in test testFailIfAlreadyExists (7.3% fail rate in 55 executions)
• [main] 2 failures in step part1 (11.1% fail rate in 18 executions)
• [main] 2 failures in step part-1 (5.9% fail rate in 34 executions)
• [main] 2 failures in pipeline elasticsearch-intake (11.1% fail rate in 18 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (5.9% fail rate in 34 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 4 failures in test testFailIfAlreadyExists (7.3% fail rate in 55 executions)
• [main] 2 failures in step part1 (11.1% fail rate in 18 executions)
• [main] 2 failures in step part-1 (5.9% fail rate in 34 executions)
• [main] 2 failures in pipeline elasticsearch-intake (11.1% fail rate in 18 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (5.9% fail rate in 34 executions)

Build Scans:

• elasticsearch-intake #28390 / part1
• elasticsearch-pull-request #91767 / part-1
• elasticsearch-pull-request #91750 / part-1
• elasticsearch-intake #28355 / part1

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DiannaHohensee]

This test went in just today (commit), so it's brand new. Looks like an external contributor (PR).

[DiannaHohensee]

The test is failing here, on the initial write, not the significant part of the test that verifies that an overwrite fails. So seems like a test setup problem, not the new functionality.

[DaveCTurner]

not the significant part of the test that verifies that an overwrite fails

Hmm no, the initial write is testing that a race between several competing writes must fail all but one of them, it's still significant. However my hunch is just that the 1-second wait for the tasks to complete is too short in CI, I thought I'd mentioned that in a review but I missed in my final pass that it hadn't been addressed.

I opened #134716 to rework this test a bit.