[CI] OpenAiServiceUpgradeIT testOpenAiCompletions {upgradedNodes=2} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #5442 / 8.14.3_bwc
• elasticsearch-periodic #5437 / 8.14.3_bwc
• elasticsearch-periodic #5372 / 8.14.3_bwc

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:rolling-upgrade:v8.14.3#bwcTest" -Dtests.class="org.elasticsearch.xpack.application.OpenAiServiceUpgradeIT" -Dtests.method="testOpenAiCompletions {upgradedNodes=1}" -Dtests.seed=A4144B90042457D0 -Dtests.bwc=true -Dtests.locale=fr-PM -Dtests.timezone=America/Juneau -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.NullPointerException: Cannot invoke "java.util.List.addAll(java.util.Collection)" because "configs" is null

Issue Reasons:

• [8.x] 3 consecutive failures in step 8.14.3_bwc
• [8.x] 3 failures in test testOpenAiCompletions {upgradedNodes=2} (1.9% fail rate in 155 executions)
• [8.x] 3 failures in step 8.14.3_bwc (100.0% fail rate in 3 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic (100.0% fail rate in 3 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[maxhniebergall]

seeing the following message in the error dashboard:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:40811], URI [_inference/completion/old-cluster-completions], status line [HTTP/1.1 400 Bad Request]
{"error":"no handler found for uri [_inference/completion/old-cluster-completions] and method [GET]"}

[maxhniebergall]

I think the root cause here is that the tests are no longer running in the order we are expecting (i.e., the oldCluster, then the mixedCluster, then the upgradedCluster).

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 13 consecutive failures in step 8.14.3_bwc
• [8.x] 13 failures in test testOpenAiCompletions {upgradedNodes=2} (2.1% fail rate in 606 executions)
• [8.x] 13 failures in step 8.14.3_bwc (100.0% fail rate in 13 executions)
• [8.x] 12 failures in pipeline elasticsearch-periodic (100.0% fail rate in 12 executions)

Build Scans:

• elasticsearch-periodic #5303 / 8.14.3_bwc
• elasticsearch-pull-request #45409 / 8.14.3_bwc
• elasticsearch-periodic #5297 / 8.14.3_bwc
• elasticsearch-periodic #5291 / 8.14.3_bwc
• elasticsearch-periodic #5285 / 8.14.3_bwc
• elasticsearch-periodic #5279 / 8.14.3_bwc
• elasticsearch-periodic #5273 / 8.14.3_bwc
• elasticsearch-periodic #5267 / 8.14.3_bwc
• elasticsearch-periodic #5261 / 8.14.3_bwc
• elasticsearch-periodic #5255 / 8.14.3_bwc

[maxhniebergall]

All of the UpgradeIT tests that started failing around the same time are on 8.x.

[maxhniebergall]

Experimental testing shows these upgrade failures were caused by this PR, but the reason this PR causes these failures is still unknown #118105