[CI] MatchOnlyTextRollingUpgradeIT testIndexing {upgradedNodes=3} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #29121 / bc-upgrade-tests-part2
• elasticsearch-intake #29123 / bc-upgrade-tests-part2
• elasticsearch-intake #29122 / bc-upgrade-tests-part2
• elasticsearch-intake #29120 / bc-upgrade-tests-part2
• elasticsearch-intake #29119 / bc-upgrade-tests-part2
• elasticsearch-intake #29118 / bc-upgrade-tests-part2
• elasticsearch-intake #29114 / bc-upgrade-tests-part2
• elasticsearch-intake #29112 / bc-upgrade-tests-part2
• elasticsearch-intake #29111 / bc-upgrade-tests-part2
• elasticsearch-intake #29110 / bc-upgrade-tests-part2

Reproduction Line:

./gradlew ":x-pack:plugin:logsdb:qa:rolling-upgrade:bcUpgradeTest" -Dtests.class="org.elasticsearch.upgrades.MatchOnlyTextRollingUpgradeIT" -Dtests.method="testIndexing {upgradedNodes=3}" -Dtests.seed=CAEC3EC19B588028 -Dtests.bwc.main.version=9.2.0-SNAPSHOT -Dtests.bwc.refspec.main=1d17ce6c97775e896bc45920ced866ab08b77880 -Dtests.locale=asa-TZ -Dtests.timezone=Indian/Cocos -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.RuntimeException: An error occurred while checking cluster 'test-cluster' status.

Issue Reasons:

• [main] 3 consecutive failures in test testIndexing {upgradedNodes=3}
• [main] 22 consecutive failures in step bc-upgrade-tests-part2
• [main] 2 consecutive failures in step 8.19.4_part_2_bwc-snapshots
• [main] 4 consecutive failures in pipeline elasticsearch-intake
• [main] 38 failures in test testIndexing {upgradedNodes=3} (4.0% fail rate in 956 executions)
• [main] 30 failures in step bc-upgrade-tests-part2 (41.7% fail rate in 72 executions)
• [main] 3 failures in step pr-upgrade-part-2 (1.8% fail rate in 168 executions)
• [main] 2 failures in step 8.19.4_part_2_bwc-snapshots (100.0% fail rate in 2 executions)
• [main] 30 failures in pipeline elasticsearch-intake (40.5% fail rate in 74 executions)
• [main] 4 failures in pipeline elasticsearch-pull-request (2.2% fail rate in 178 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in step bc-upgrade-tests-part2
• [main] 5 consecutive failures in step 8.19.4_part_2_bwc-snapshots
• [main] 5 consecutive failures in step 9.1.4_part_2_bwc-snapshots
• [main] 2 consecutive failures in pipeline elasticsearch-intake
• [main] 22 failures in test testIndexing {upgradedNodes=3} (2.4% fail rate in 915 executions)
• [main] 2 failures in step bc-upgrade-tests-part2 (4.7% fail rate in 43 executions)
• [main] 4 failures in step 9.0.7_part_2_bwc-snapshots (66.7% fail rate in 6 executions)
• [main] 5 failures in step 8.19.4_part_2_bwc-snapshots (100.0% fail rate in 5 executions)
• [main] 5 failures in step 9.1.4_part_2_bwc-snapshots (100.0% fail rate in 5 executions)
• [main] 2 failures in pipeline elasticsearch-intake (4.5% fail rate in 44 executions)
• [main] 5 failures in pipeline elasticsearch-pull-request (2.7% fail rate in 183 executions)

Build Scans:

• elasticsearch-intake #28941 / bc-upgrade-tests-part2
• elasticsearch-intake #28932 / bc-upgrade-tests-part2
• elasticsearch-periodic-java-ea #31 / 9.1.5_bwc
• elasticsearch-periodic-java-ea #31 / 9.0.8_bwc
• elasticsearch-periodic-java-ea #31 / 8.19.5_bwc
• elasticsearch-periodic-java-ea #31 / 9.0.8_matrix-bwc
• elasticsearch-periodic-java-ea #31 / 8.19.5_matrix-bwc
• elasticsearch-periodic-java-ea #31 / 9.1.5_matrix-bwc
• elasticsearch-pull-request #94114 / 8.19.4_part_2_bwc-snapshots
• elasticsearch-pull-request #94114 / 9.1.4_part_2_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 12 failures in test testIndexing {upgradedNodes=3} (1.6% fail rate in 742 executions)
• [main] 9 failures in step bc-upgrade-tests-part2 (17.6% fail rate in 51 executions)
• [main] 2 failures in step pr-upgrade-part-2 (1.6% fail rate in 122 executions)
• [main] 9 failures in pipeline elasticsearch-intake (17.3% fail rate in 52 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (2.3% fail rate in 130 executions)

Build Scans:

• elasticsearch-intake #29022 / bc-upgrade-tests-part2
• elasticsearch-pull-request #94725 / pr-upgrade-part-2
• elasticsearch-intake #29006 / bc-upgrade-tests-part2
• elasticsearch-intake #28988 / bc-upgrade-tests-part2
• elasticsearch-intake #28979 / bc-upgrade-tests-part2
• elasticsearch-pull-request #94592 / pr-upgrade-part-2
• elasticsearch-intake #28969 / bc-upgrade-tests-part2
• elasticsearch-pull-request #94478 / 8.19.4_part_2_bwc-snapshots
• elasticsearch-intake #28938 / bc-upgrade-tests-part2
• elasticsearch-intake #28940 / bc-upgrade-tests-part2