[CI] MultiClusterSpecIT test {categorize.Categorize} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #5390 / 8.15.5_bwc
• elasticsearch-periodic #5307 / openjdk22_8.16.2_java-matrix-bwc

Reproduction Line:

./gradlew ":x-pack:plugin:esql:qa:server:multi-clusters:v8.15.5#bwcTest" -Dtests.class="org.elasticsearch.xpack.esql.ccq.MultiClusterSpecIT" -Dtests.method="test {categorize.Categorize}" -Dtests.seed=92D3937922D231E2 -Dtests.bwc=true -Dtests.locale=dyo-SN -Dtests.timezone=US/Samoa -Druntime.java=23

Applicable branches:
8.16

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Data mismatch:
row 0 column 0:a list containing
row 0 column 0:0: expected "3" but was "1"
row 0 column 1:a list containing
row 0 column 1:0: expected "[Connected to 10.1.0.1, Connected to 10.1.0.2, Connected to 10.1.0.3]" but was "Disconnected"
row 1 column 1:a list containing
row 1 column 1:0: expected "Connection error" but was "[Connected to 10.1.0.1, Connected to 10.1.0.2, Connected to 10.1.0.3]"
row 2 column 0:a list containing
row 2 column 0:0: expected "1" but was "3"
row 2 column 1:a list containing
row 2 column 1:0: expected "Disconnected" but was "Connection error"

Issue Reasons:

• [8.16] 2 failures in test test {categorize.Categorize} (0.4% fail rate in 506 executions)
• [8.16] 2 failures in pipeline elasticsearch-periodic (12.5% fail rate in 16 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[alex-spies]

If I read this correctly, it looks like the categories are assigned different category ids in this CCQ scenario. Which probably can happen, because the category ids depend on which strings the categorizer sees first. So instead of the expected

3          | [Connected to 10.1.0.1, Connected to 10.1.0.2, Connected to 10.1.0.3] | 0
3          | [Connection error]                                                    | 1
1          | [Disconnected]                                                        | 2

I think it sees

1          | [Disconnected]                                                        | 0
3          | [Connected to 10.1.0.1, Connected to 10.1.0.2, Connected to 10.1.0.3] | 1
3          | [Connection error]                                                    | 2

Maybe let's mute this pro-actively on 8.16 so we don't produce failures on 8.16.

@jan-elastic , should we just mute categorize.Categorize on org.elasticsearch.xpack.esql.ccq.MultiClusterSpecIT? We could also backport a ignoreOrder: true, instead. no we can't, it's not the ordering of the result but the order of documents the categorizer sees.