[CI] FullClusterRestartIT testSoftDeletes {cluster=UPGRADED} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #29712 / 10.0.0_lucene-compat
• elasticsearch-intake #29557 / 10.0.0_lucene-compat

Reproduction Line:

./gradlew ":qa:full-cluster-restart:luceneBwcTest" -Dtests.class="org.elasticsearch.upgrades.FullClusterRestartIT" -Dtests.method="testSoftDeletes {cluster=UPGRADED}" -Dtests.seed=7CC96514EE2FB2BE -Dtests.bwc.main.version=9.0.0 -Dtests.bwc.refspec.main=10352e57d85505984582616e1e38530d3ec6ca59 -Dtests.locale=mi-NZ -Dtests.timezone=Pacific/Pago_Pago -Druntime.java=25

Applicable branches:
9.2

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.RuntimeException: An error occurred while checking cluster 'test-cluster' status.

Issue Reasons:

• [9.2] 2 failures in test testSoftDeletes {cluster=UPGRADED} (0.4% fail rate in 519 executions)
• [9.2] 2 failures in step 10.0.0_lucene-compat (28.6% fail rate in 7 executions)
• [9.2] 2 failures in pipeline elasticsearch-intake (6.9% fail rate in 29 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in step 10.0.0_lucene-compat
• [main] 2 failures in test testSoftDeletes {cluster=UPGRADED} (0.2% fail rate in 1000 executions)
• [main] 2 failures in step 10.0.0_lucene-compat (10.0% fail rate in 20 executions)
• [main] 2 failures in pipeline elasticsearch-intake (4.4% fail rate in 45 executions)

Build Scans:

• elasticsearch-intake #29526 / 10.0.0_lucene-compat
• elasticsearch-intake #29510 / 10.0.0_lucene-compat

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticsearchmachine]

This has been muted on branch 9.2

Mute Reasons:

• [9.2] 2 failures in test testSoftDeletes {cluster=UPGRADED} (0.4% fail rate in 519 executions)
• [9.2] 2 failures in step 10.0.0_lucene-compat (28.6% fail rate in 7 executions)
• [9.2] 2 failures in pipeline elasticsearch-intake (6.9% fail rate in 29 executions)

Build Scans:

• elasticsearch-intake #29712 / 10.0.0_lucene-compat
• elasticsearch-intake #29557 / 10.0.0_lucene-compat