[CI] FullClusterRestartIT testWatcher {cluster=UPGRADED} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #5755 / 7.1.1_bwc
• elasticsearch-pull-request #49316 / 7.4.2_bwc
• elasticsearch-periodic #5735 / 7.4.2_bwc
• elasticsearch-periodic #5735 / 7.1.1_bwc
• elasticsearch-periodic #5727 / 7.1.1_bwc
• elasticsearch-periodic #5727 / 7.4.2_bwc
• elasticsearch-periodic #5722 / 7.2.1_bwc
• elasticsearch-periodic #5717 / 7.0.1_bwc
• elasticsearch-periodic #5702 / 7.3.2_bwc

Reproduction Line:

./gradlew ":x-pack:qa:full-cluster-restart:v7.1.1#bwcTest" -Dtests.class="org.elasticsearch.xpack.restart.FullClusterRestartIT" -Dtests.method="testWatcher {cluster=UPGRADED}" -Dtests.seed=CD5C215BDFD2CF1B -Dtests.bwc=true -Dtests.locale=rof -Dtests.timezone=Australia/Currie -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.RuntimeException: An error occurred while checking cluster 'test-cluster' status.

Issue Reasons:

• [8.x] 3 consecutive failures in step 7.1.1_bwc
• [8.x] 3 consecutive failures in step 7.4.2_bwc
• [8.x] 9 failures in test testWatcher {cluster=UPGRADED} (1.0% fail rate in 884 executions)
• [8.x] 3 failures in step 7.1.1_bwc (100.0% fail rate in 3 executions)
• [8.x] 3 failures in step 7.4.2_bwc (100.0% fail rate in 3 executions)
• [8.x] 6 failures in pipeline elasticsearch-periodic (60.0% fail rate in 10 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[mark-vieira]

This looks like a legit BWC issue on 8.x when doing a full cluster restart upgrade from 7.x.

[2024-12-23T23:42:35,100][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [test-cluster-0] fatal error in thread [elasticsearch[test-cluster-0][watcher][T#4]], exiting java.lang.AssertionError	
	at org.elasticsearch.xcore@8.18.0-SNAPSHOT/org.elasticsearch.xpack.core.watcher.execution.WatchExecutionContext.start(WatchExecutionContext.java:157)	
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:520)	
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:341)	
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$6(ExecutionService.java:439)	
	at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:665)