[CI] CrossClusterQueryWithFiltersIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #9420 / oraclelinux-9_platform-support-unix
• elasticsearch-periodic-platform-support #9329 / rhel-9_platform-support-unix
• elasticsearch-periodic-platform-support #9301 / amazonlinux-2_platform-support-aws
• elasticsearch-intake #25335 / part3

Reproduction Line:

undefined

Applicable branches:
8.19

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

• [8.19] 4 failures in class org.elasticsearch.xpack.esql.action.CrossClusterQueryWithFiltersIT (0.5% fail rate in 767 executions)
• [8.19] 3 failures in pipeline elasticsearch-periodic-platform-support (14.3% fail rate in 21 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[astefan]

@smalyshev can you take a look at this one?
It doesn't repro for me, even after hundreds of executions. Somehow one of the nodes cannot be reached and there is a connect_exception raised, but it's not clear to me why. Seems to be a shutdown cluster that is not synchronized with the test itself, but it's just a guess.

[smalyshev]

I think I've seen it before: #127332 Somehow sometimes, very rarely, one shard gets skipped. Still not sure why. I'll take another look on it - it's probably some race with something, but I still didn't figure out with what.

[smalyshev]

Yep I checked the errors and this is the full duplicate of #127332