[CI] EnterpriseSearchRestIT test {p0=entsearch/90_behavioral_analytics_event_post/Post analytics event - Unknown event field} failing

[masseyke]

Build scan:
https://gradle-enterprise.elastic.co/s/i4c6rbbsw2zwo/tests/:x-pack:plugin:ent-search:qa:rest:yamlRestTest/org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT/test%20%7Bp0=entsearch%2F90_behavioral_analytics_event_post%2FPost%20analytics%20event%20-%20Unknown%20event%20field%7D

Reproduction line:

./gradlew ':x-pack:plugin:ent-search:qa:rest:yamlRestTest' --tests "org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT.test {p0=entsearch/90_behavioral_analytics_event_post/Post analytics event - Unknown event field}" -Dtests.seed=1C756D99D9475A47 -Dtests.locale=pt-BR -Dtests.timezone=America/Cayenne -Druntime.java=20 -Dtests.fips.enabled=true

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT&tests.test=test%20%7Bp0%3Dentsearch/90_behavioral_analytics_event_post/Post%20analytics%20event%20-%20Unknown%20event%20field%7D

Failure excerpt:

java.lang.AssertionError: Failure at [entsearch/90_behavioral_analytics_event_post:2]: expected [2xx] status code but api [search_application.put_behavioral_analytics] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"analytics collection [my-test-analytics-collection] already exists","stack_trace":"org.elasticsearch.ResourceAlreadyExistsException: analytics collection [my-test-analytics-collection] already exists\n\tat org.elasticsearch.application@8.9.0-SNAPSHOT/org.elasticsearch.xpack.application.analytics.AnalyticsCollectionService.lambda$putAnalyticsCollection$1(AnalyticsCollectionService.java:91)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:166)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:126)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:131)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:166)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.lambda$notifyListenerFailed$1(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.executeInContext(AllocationActionListener.java:58)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.notifyListenerFailed(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener$1.onFailure(AllocationActionListener.java:84)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:75)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:357)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:357)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\n"}],"type":"resource_already_exists_exception","reason":"analytics collection [my-test-analytics-collection] already exists","stack_trace":"org.elasticsearch.ResourceAlreadyExistsException: analytics collection [my-test-analytics-collection] already exists\n\tat org.elasticsearch.application@8.9.0-SNAPSHOT/org.elasticsearch.xpack.application.analytics.AnalyticsCollectionService.lambda$putAnalyticsCollection$1(AnalyticsCollectionService.java:91)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:166)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:126)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:131)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:166)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.lambda$notifyListenerFailed$1(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.executeInContext(AllocationActionListener.java:58)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.notifyListenerFailed(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener$1.onFailure(AllocationActionListener.java:84)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:75)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:357)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:357)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\n"},"status":400}]

  at __randomizedtesting.SeedInfo.seed([1C756D99D9475A47:9421524377BB37BF]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:572)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:515)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

  Caused by: java.lang.AssertionError: expected [2xx] status code but api [search_application.put_behavioral_analytics] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"analytics collection [my-test-analytics-collection] already exists","stack_trace":"org.elasticsearch.ResourceAlreadyExistsException: analytics collection [my-test-analytics-collection] already exists\n\tat org.elasticsearch.application@8.9.0-SNAPSHOT/org.elasticsearch.xpack.application.analytics.AnalyticsCollectionService.lambda$putAnalyticsCollection$1(AnalyticsCollectionService.java:91)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:166)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:126)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:131)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:166)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.lambda$notifyListenerFailed$1(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.executeInContext(AllocationActionListener.java:58)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.notifyListenerFailed(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener$1.onFailure(AllocationActionListener.java:84)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:75)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:357)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:357)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\n"}],"type":"resource_already_exists_exception","reason":"analytics collection [my-test-analytics-collection] already exists","stack_trace":"org.elasticsearch.ResourceAlreadyExistsException: analytics collection [my-test-analytics-collection] already exists\n\tat org.elasticsearch.application@8.9.0-SNAPSHOT/org.elasticsearch.xpack.application.analytics.AnalyticsCollectionService.lambda$putAnalyticsCollection$1(AnalyticsCollectionService.java:91)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:166)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:126)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:131)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:166)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.lambda$notifyListenerFailed$1(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.executeInContext(AllocationActionListener.java:58)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.notifyListenerFailed(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener$1.onFailure(AllocationActionListener.java:84)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:75)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:357)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:357)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\n"},"status":400}]

    at org.junit.Assert.fail(Assert.java:88)
    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:373)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:552)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:515)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke(Method.java:578)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1623)

[joegallo]

Two more of the same (well, approximately the same):

https://gradle-enterprise.elastic.co/s/5rc56wpdanoeu/tests/:x-pack:plugin:ent-search:qa:rest:yamlRestTest/org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT/test%20%7Bp0=entsearch%2F90_behavioral_analytics_event_post%2FPost%20search_click%20analytics%20event%20-%20Document%20Only%7D?top-execution=1

https://gradle-enterprise.elastic.co/s/iveqkati52lue/tests/:x-pack:plugin:ent-search:qa:rest:yamlRestTest/org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT/test%20%7Bp0=entsearch%2F90_behavioral_analytics_event_post%2FPost%20page_view%20analytics%20event%20-%20Missing%20session.id%7D?top-execution=1

And another nearby test:

https://gradle-enterprise.elastic.co/s/y3rgx6x5mkwrw/tests/:x-pack:plugin:ent-search:qa:rest:yamlRestTest/org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT/test%20%7Bp0=entsearch%2F80_behavioral_analytics_delete%2FDelete%20Analytics%20Collection%20-%20Analytics%20Collection%20does%20not%20exist%7D?top-execution=1

[joegallo]

https://gradle-enterprise.elastic.co/scans/tests?search.buildOutcome=failure&search.startTimeMax=1683345600000&search.startTimeMin=1682481600000&search.tags=CI&search.timeZoneId=America/New_York&tests.container=org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT

Useful graph of failures starting yesterday and extending through the end of next week, so we can see how frequently this fails on CI.

[joegallo]

Just running the overall test suite in a loop seems pretty reliable to me. I got 7 failures out of 10 runs with this:

for i in {0..9}; do ./gradlew ':x-pack:plugin:ent-search:qa:rest:yamlRestTest' --tests "org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT.test {p0=entsearch/90_behavioral_analytics_event_post/*"; done

[mayya-sharipova]

Similar failure on build for my PR:

./gradlew ':x-pack:plugin:ent-search:qa:rest:yamlRestTest' --tests "org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT.test {p0=entsearch/90_behavioral_analytics_event_post/Post search_click analytics event - Document Only}" -Dtests.seed=EDDF5584110E2E95 -Dtests.locale=es-ES -Dtests.timezone=Atlantic/Madeira -Druntime.java=20

[joegallo]

No luck with reproducing for that other nearby test file, though. It passed 100 times in a row for me.

for i in {0..99}; do ./gradlew ':x-pack:plugin:ent-search:qa:rest:yamlRestTest' --tests "org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT.test {p0=entsearch/80_behavioral_analytics_delete/*}"; done

[ywangd]

It semes the issue (#95645) that I raised is likely a duplicate of this one. I think the failure is more likely to reproduce when running the entire test suite together, e.g.

./gradlew ':x-pack:plugin:ent-search:qa:rest:yamlRestTest' --tests "org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT.test {p0=entsearch/*}" -Dtests.seed=E0FAA98B42596376 -Dtests.locale=nb-NO -Dtests.timezone=Africa/Ceuta -Druntime.java=20

There are some interaction between tests. More specifically it seems some tests did not clean up after them which led to "resource_already_exists_exception" for others.

[pxsalehi]

More failure of this: https://gradle-enterprise.elastic.co/s/nloki2spvuhjc/tests/:x-pack:plugin:ent-search:qa:rest:yamlRestTest/org.elasticsearch.xpack.entsearch.EnterpriseSearchRestIT/test%20%7Bp0=entsearch%2F90_behavioral_analytics_event_post%2FPost%20analytics%20event%20-%20Unknown%20event%20field%7D?top-execution=1