[CI] CohereServiceUpgradeIT testCohereEmbeddings {upgradedNodes=2} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #16936 / 8.19.0_bwc-snapshots
• elasticsearch-intake #16934 / 8.19.0_bwc-snapshots
• elasticsearch-intake #16932 / 8.19.0_bwc-snapshots

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:rolling-upgrade:v8.19.0#bwcTest" -Dtests.class="org.elasticsearch.xpack.application.CohereServiceUpgradeIT" -Dtests.method="testRerank {upgradedNodes=3}" -Dtests.seed=9DE645285488F16E -Dtests.bwc=true -Dtests.locale=ka-Geor-GE -Dtests.timezone=Africa/Banjul -Druntime.java=23

Applicable branches:
9.0

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.Exception: Test abandoned because suite timeout was reached.

Issue Reasons:

• [9.0] 3 consecutive failures in step 8.19.0_bwc-snapshots
• [9.0] 3 failures in test testCohereEmbeddings {upgradedNodes=2} (5.8% fail rate in 52 executions)
• [9.0] 3 failures in step 8.19.0_bwc-snapshots (18.8% fail rate in 16 executions)
• [9.0] 3 failures in pipeline elasticsearch-intake (25.0% fail rate in 12 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 9.0

Mute Reasons:

• [9.0] 2 consecutive failures in step 8.19.0_bwc-snapshots
• [9.0] 2 consecutive failures in pipeline elasticsearch-intake
• [9.0] 2 failures in test testCohereEmbeddings {upgradedNodes=2} (4.2% fail rate in 48 executions)
• [9.0] 2 failures in step 8.19.0_bwc-snapshots (13.3% fail rate in 15 executions)
• [9.0] 2 failures in pipeline elasticsearch-intake (18.2% fail rate in 11 executions)

Build Scans:

• elasticsearch-intake #16934 / 8.19.0_bwc-snapshots
• elasticsearch-intake #16932 / 8.19.0_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[tlrx]

I suspect this test failed because of the reason I explain in this #121365 (comment)

[jonathan-buttner]

Looks like the fix was merged, I'll create a PR to unmute the upgrade tests for the inference plugin.