[CI] DefaultEndPointsIT testMultipleInferencesTriggeringDownloadAndDeploy failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #30570 / part2
• elasticsearch-intake #30569 / part2
• elasticsearch-pull-request #101022 / part-2
• elasticsearch-pull-request #101021 / part-2
• elasticsearch-pull-request #101015 / part-2
• elasticsearch-pull-request #101012 / part-2
• elasticsearch-pull-request #100995 / part-2
• elasticsearch-pull-request #100992 / part-2
• elasticsearch-pull-request #100973 / part-2
• elasticsearch-pull-request #100961 / part-2

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:inference-service-tests:javaRestTest" --tests "org.elasticsearch.xpack.inference.DefaultEndPointsIT.testMultipleInferencesTriggeringDownloadAndDeploy" -Dtests.seed=BA33C76A56DA53F8 -Dtests.locale=ks-IN -Dtests.timezone=Asia/Dubai -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: [org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:33287], URI [_inference/.elser-2-elasticsearch?timeout=120s], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":null}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-inference-000005","node":null,"reason":{"type":"no_shard_available_action_exception","reason":null}}]},"status":503}]
Expected: an empty collection
     but: <[org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:33287], URI [_inference/.elser-2-elasticsearch?timeout=120s], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":null}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"sha
[truncated]

Issue Reasons:

• [main] 2 consecutive failures in test testMultipleInferencesTriggeringDownloadAndDeploy
• [main] 2 consecutive failures in step part2
• [main] 2 consecutive failures in step part-2
• [main] 2 consecutive failures in pipeline elasticsearch-intake
• [main] 2 consecutive failures in pipeline elasticsearch-pull-request
• [main] 14 failures in test testMultipleInferencesTriggeringDownloadAndDeploy (19.2% fail rate in 73 executions)
• [main] 2 failures in step part2 (16.7% fail rate in 12 executions)
• [main] 12 failures in step part-2 (21.1% fail rate in 57 executions)
• [main] 2 failures in pipeline elasticsearch-intake (16.7% fail rate in 12 executions)
• [main] 11 failures in pipeline elasticsearch-pull-request (20.8% fail rate in 53 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 9 failures in test testMultipleInferencesTriggeringDownloadAndDeploy (2.3% fail rate in 394 executions)
• [main] 7 failures in step part-2 (3.5% fail rate in 202 executions)
• [main] 8 failures in pipeline elasticsearch-pull-request (4.0% fail rate in 202 executions)

Build Scans:

• elasticsearch-intake #13195 / part2
• elasticsearch-pull-request #42333 / part-2
• elasticsearch-pull-request #42321 / part-2
• elasticsearch-pull-request #42242 / part-2
• elasticsearch-pull-request #42200 / part-2
• elasticsearch-pull-request #42168 / part-2
• elasticsearch-pull-request #42142 / part-2
• elasticsearch-pull-request #41998 / part-2
• elasticsearch-pull-request #41915 / part-2-fips

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

#65846 again

The test makes multiple concurrent calls to the preconfigured inference endpoints, depending on the timing many of those calls will cause writes to .secrets-inference or .inference auto creating the index. Once the index is in the process of being created then there is a window when writes will fail because the index is properly initialised.

Low risk as this is just a test issue and not a realistic real world use case. One option would be to create the index by write something to it before starting the test proper

[elasticsearchmachine]

This has been muted on branch 8.17

Mute Reasons:

• [8.17] 2 failures in test testMultipleInferencesTriggeringDownloadAndDeploy (1.4% fail rate in 144 executions)

Build Scans:

• elasticsearch-periodic-platform-support #5046 / almalinux-8-aarch64_checkpart2_platform-support-arm
• elasticsearch-pull-request #43065 / part-2

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 2 failures in test testMultipleInferencesTriggeringDownloadAndDeploy (0.3% fail rate in 594 executions)

Build Scans:

• elasticsearch-periodic #5087 / openjdk21_checkpart2_java-matrix
• elasticsearch-pull-request #42926 / part-2