[CI] ClusterRerouteIT testDelayWithALargeAmountOfShards failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #11354 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-periodic-platform-support #11346 / windows-2019_checkpart1_platform-support-windows

Reproduction Line:

gradlew ":server:internalClusterTest" --tests "org.elasticsearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards" -Dtests.seed=A74D9E2714A276E9 -Dtests.locale=lo-Laoo-LA -Dtests.timezone=Atlantic/Madeira -Druntime.java=25

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: timed out waiting for green state

Issue Reasons:

• [8.19] 2 failures in test testDelayWithALargeAmountOfShards (2.9% fail rate in 69 executions)
• [8.19] 2 failures in pipeline elasticsearch-periodic-platform-support (66.7% fail rate in 3 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[ywangd]

The 4 failures in the past 30 days are all on Windows platform. It is a bit odd that it started to fail since about 6 days ago. But overall I suspect it is due to slowness of the Windows platform. Note we already exclude WindowsFS for the test class specifically due to this test method (#52488). I wonder whether the real Windows file system has similar performance issue.

There is nothing particular interesting in the failure logs. The cluster was its way to recovery. The process was just slow as indicated by 2-3 seconds gaps between the marking unavailable shards as stale messages. The recovery was almost completed (only 1 more shard to recover) when the test timed out (2min). Therefore I think we can either skip this test when running on Windows or increase the timeout when running on Windows.