[CI] WriteLoadConstraintMonitorTests testRerouteIsNotCalledWhenNoNodesAreHotSpotting failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #11532 / single-processor-node-tests
• elasticsearch-pull-request #107137 / part-1
• elasticsearch-pull-request #106494 / part-1

Reproduction Line:

./gradlew ":server:test" --tests "org.elasticsearch.cluster.routing.allocation.WriteLoadConstraintMonitorTests.testRerouteIsNotCalledWhenNoNodesAreHotSpotting" -Dtests.seed=5EDE1F35119342F2 -Dtests.configure_test_clusters_with_one_processor=true -Dtests.locale=en-ES -Dtests.timezone=Asia/Thimphu -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: expected to see don't reroute due to no nodes hot-spotting but did not
Expected: <0L>
     but: was <1L>

Issue Reasons:

• [main] 3 failures in test testRerouteIsNotCalledWhenNoNodesAreHotSpotting (0.4% fail rate in 749 executions)
• [main] 2 failures in step part-1 (0.5% fail rate in 409 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (0.5% fail rate in 406 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testRerouteIsNotCalledWhenNoNodesAreHotSpotting (0.4% fail rate in 749 executions)
• [main] 2 failures in step part-1 (0.5% fail rate in 409 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (0.5% fail rate in 406 executions)

Build Scans:

• elasticsearch-periodic #11532 / single-processor-node-tests
• elasticsearch-pull-request #107137 / part-1
• elasticsearch-pull-request #106494 / part-1

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DiannaHohensee]

Have not verified, but my first thought is that likely the monitor ran during set-up before the real test logic had a chance to run. I see the monitor interval settings are random 1 to 30 seconds for testing: I would expect the monitor running on its own to be disabled, by setting a big interval, and the tests to do manual cluster info refreshes as needed.