[CI] MultiClusterSpecIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #28285 / 9.2.0_bwc-snapshots
• elasticsearch-intake #28283 / 8.19.4_bwc-snapshots
• elasticsearch-intake #28283 / 9.2.0_bwc-snapshots
• elasticsearch-pull-request #91413 / 9.2.0_part_3_bwc-snapshots
• elasticsearch-intake #28282 / 9.0.7_bwc-snapshots
• elasticsearch-intake #28282 / 9.1.4_bwc-snapshots
• elasticsearch-intake #28280 / 9.1.4_bwc-snapshots
• elasticsearch-intake #28280 / 9.0.7_bwc-snapshots
• elasticsearch-intake #28280 / 9.2.0_bwc-snapshots
• elasticsearch-pull-request #91410 / 9.1.4_part_3_bwc-snapshots

Reproduction Line:

./gradlew ":x-pack:plugin:esql:qa:server:multi-clusters:v9.2.0#newToOld" -Dtests.class="org.elasticsearch.xpack.esql.ccq.MultiClusterSpecIT" -Dtests.method="test {csv-spec:tbucket.TbucketByOneYearDuration}" -Dtests.seed=9555BF72012DAE31 -Dtests.bwc=true -Dtests.locale=en-US -Dtests.timezone=Asia/Colombo -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [PUT], host [http://[::1]:40549], URI [/languages_lookup_non_unique_key], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [languages_lookup_non_unique_key/duwetMngTOenOc1FNKCkGw] already exists","index_uuid":"duwetMngTOenOc1FNKCkGw","index":"languages_lookup_non_unique_key"}],"type":"resource_already_exists_exception","reason":"index [languages_lookup_non_unique_key/duwetMngTOenOc1FNKCkGw] already exists","index_uuid":"duwetMngTOenOc1FNKCkGw","index":"languages_lookup_non_unique_key"},"status":400}

Issue Reasons:

• [main] 2 consecutive failures in step 9.1.4_bwc-snapshots
• [main] 23 failures in class org.elasticsearch.xpack.esql.ccq.MultiClusterSpecIT (2.3% fail rate in 1000 executions)
• [main] 4 failures in step 9.2.0_bwc-snapshots (21.1% fail rate in 19 executions)
• [main] 3 failures in step 9.0.7_bwc-snapshots (18.8% fail rate in 16 executions)
• [main] 2 failures in step 9.2.0_part_3_bwc-snapshots (3.1% fail rate in 65 executions)
• [main] 4 failures in step 8.19.4_bwc-snapshots (25.0% fail rate in 16 executions)
• [main] 4 failures in step 9.1.4_bwc-snapshots (26.7% fail rate in 15 executions)
• [main] 10 failures in pipeline elasticsearch-intake (55.6% fail rate in 18 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (4.5% fail rate in 66 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[astefan]

I don't think I've seen a failure to create indices because they already exists so far. Does the new loading in parallel enhancement have anything to do with this @julian-elastic? Can you please have a look?

The full error:

java.lang.AssertionError: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Failed to create indices in parallel
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: Failed to create indices in parallel
   > java.lang.RuntimeException: Failed to create indices in parallel
      > org.elasticsearch.client.ResponseException: method [PUT], host [http://[::1]:37541], URI [/languages_lookup_non_unique_key], status line [HTTP/1.1 400 Bad Request]
        {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [languages_lookup_non_unique_key/-68F6wM6RUW3ETkA33oXHA] already exists","index_uuid":"-68F6wM6RUW3ETkA33oXHA","index":"languages_lookup_non_unique_key"}],"type":"resource_already_exists_exception","reason":"index [languages_lookup_non_unique_key/-68F6wM6RUW3ETkA33oXHA] already exists","index_uuid":"-68F6wM6RUW3ETkA33oXHA","index":"languages_lookup_non_unique_key"},"status":400}

[julian-elastic]

Yes, this is related to my check in here #134090. I will revert it while I work on a fix.