[CI] WriteLoadConstraintDeciderIT testCanRemainNotPreferredIsIgnoredWhenAllOtherNodesReturnNotPreferred failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #11337 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-pull-request #99952 / part-1

Reproduction Line:

gradlew ":server:internalClusterTest" --tests "org.elasticsearch.cluster.routing.allocation.decider.WriteLoadConstraintDeciderIT.testCanRemainNotPreferredIsIgnoredWhenAllOtherNodesReturnNotPreferred" -Dtests.seed=D6E11A27B3954B39 -Dtests.locale=bs -Dtests.timezone=Africa/Bujumbura -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: safeGet: listener was completed exceptionally

Issue Reasons:

• [main] 2 failures in test testCanRemainNotPreferredIsIgnoredWhenAllOtherNodesReturnNotPreferred (0.3% fail rate in 726 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testCanRemainNotPreferredIsIgnoredWhenAllOtherNodesReturnNotPreferred (0.3% fail rate in 726 executions)

Build Scans:

• elasticsearch-periodic-platform-support #11337 / windows-2022_checkpart1_platform-support-windows
• elasticsearch-pull-request #99952 / part-1

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[ywangd]

I am focusing on the test failure from the periodical CI. It failed at test setup time where we use filter allocation to ensure all shards to be on the first node. The cluster was green at the time but somehow the assertion failed. At this point, no actual test code was executed. It's probably still a test issue but not obvious to me. I am labelling it as medium-risk in case it is an issue with the existing filter allocation logic or some unintended interplay with it due to recent changes.

I raised #137171 to add more logs and unmute the test to see if it reproduces on CI with more debug info.