[CI] CohereServiceMixedIT testRerank failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #11325 / 9.0.8_bwc
• elasticsearch-pull-request #99733 / pr-upgrade-part-2
• elasticsearch-pull-request #99733 / 8.19.6_part_2_bwc-snapshots
• elasticsearch-periodic #11275 / 9.0.8_bwc

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:mixed-cluster:v9.0.8#javaRestTest" -Dtests.class="org.elasticsearch.xpack.inference.qa.mixed.CohereServiceMixedIT" -Dtests.method="testRerank" -Dtests.seed=3C5A13D28573B9C6 -Dtests.bwc=true -Dtests.locale=cgg-UG -Dtests.timezone=America/Santarem -Druntime.java=25

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:39505], URI [/_features/_reset], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"timeout_exception","reason":"Unknown error occurred while updating cluster state","timed_out":true}],"type":"exception","reason":"failed to disable reset mode after state otherwise successful machine learning reset","caused_by":{"type":"timeout_exception","reason":"Unknown error occurred while updating cluster state","timed_out":true}},"status":500}

Issue Reasons:

• [main] 4 failures in test testRerank (0.4% fail rate in 999 executions)
• [main] 2 failures in step 9.0.8_bwc (18.2% fail rate in 11 executions)
• [main] 2 failures in pipeline elasticsearch-periodic (18.2% fail rate in 11 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 4 failures in test testRerank (0.4% fail rate in 999 executions)
• [main] 2 failures in step 9.0.8_bwc (18.2% fail rate in 11 executions)
• [main] 2 failures in pipeline elasticsearch-periodic (18.2% fail rate in 11 executions)

Build Scans:

• elasticsearch-periodic #11325 / 9.0.8_bwc
• elasticsearch-pull-request #99733 / pr-upgrade-part-2
• elasticsearch-pull-request #99733 / 8.19.6_part_2_bwc-snapshots
• elasticsearch-periodic #11275 / 9.0.8_bwc

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[dan-rubinstein]

Seems that the tests are failing as we are timing out while waiting for the feature reset API to complete. This seems to affect multiple MixedIT tests but Cohere has been the first to have this fail enough to mute. There is a PR which should help increase the timeout on these calls. Once this is merged we can unmute the tests.

[DaveCTurner]

Hmm I'm not convinced this is due to a genuine timeout, but the logging is unclear. I opened #137133 to extend the timeout and also increase the log verbosity.