[CI] DesiredBalanceComputerTests testDesiredBalanceShouldConvergeInABigCluster failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #28764 / part1
• elasticsearch-periodic-java-ea #27 / release-tests

Reproduction Line:

./gradlew ":server:test" --tests "org.elasticsearch.cluster.routing.allocation.allocator.DesiredBalanceComputerTests.testDesiredBalanceShouldConvergeInABigCluster" -Dtests.seed=98C8F0904B6D994E -Dtests.locale=fr-GF -Dtests.timezone=Africa/Khartoum -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Balance should converge, but exited by the iteration limit
Expected: <686804844L>
     but: was <-1L>

Issue Reasons:

• [main] 2 failures in test testDesiredBalanceShouldConvergeInABigCluster (0.8% fail rate in 253 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testDesiredBalanceShouldConvergeInABigCluster (0.8% fail rate in 253 executions)

Build Scans:

• elasticsearch-intake #28764 / part1
• elasticsearch-periodic-java-ea #27 / release-tests

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[joshua-adams-1]

The cluster created has 7 nodes and 5460 shards, and required 1090 iterations to converge. The limit was set to 1000, here.

I've started a slack thread here

I'm considering bumping this value to resolve the failed test. However, if I pick a threshold too big, then we're wasting cycles in the case of failed tests, but if the threshold isn't big enough, then we'll have another failure in future. Therefore I'm going to dig into the test and set all randomised variables to their largest possible values and see what the maximum number of iterations required for convergence was (since this should be the absolute maximum value of any combination).

[joshua-adams-1]

I've ran the test 10K+ times, and I haven't seen an occurrence needing greater than 1100 iterations to converge. Therefore, to add a safety buffer, I'm going to bump the threshold to 2000