Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Failure in org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=transform/transforms_start_stop/Test stop missing transform} #108959

Closed
thecoop opened this issue May 23, 2024 · 5 comments
Closed

[CI] Failure in org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=transform/transforms_start_stop/Test stop missing transform} #108959

thecoop opened this issue May 23, 2024 · 5 comments
Labels
:ml/Transform Transform needs:risk Requires assignment of a risk label (low, medium, blocker) Team:ML Meta label for the ML team >test-failure Triaged test failures from CI v8.14.0

Comments

@thecoop
Copy link
Member

thecoop commented May 23, 2024 •
edited

It's a bit hard to figure out what exactly happened, here, but there's lots of failures around yaml tests

Failure in [elasticsearch / periodic / platform-support / 8.14 / ubuntu-2004-aarch64 / checkPart3 / platform-support-arm](https://gradle-enterprise.elastic.co/s/xw4dpy6yvnwia) - https://gradle-enterprise.elastic.co/s/xw4dpy6yvnwia

First failure reproduction ./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline}" -Dtests.seed=858DD2DE27314F95 -Dtests.locale=zh-Hant-HK -Dtests.timezone=Africa/Douala -Druntime.java=22

Stack trace

org.elasticsearch.xpack.test.rest.XPackRestIT > test {p0=transform/transforms_start_stop/Test stop missing transform} FAILED	
    java.lang.AssertionError: Failure at [transform/transforms_start_stop:2]: expected [2xx] status code but api [indices.create] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [airline-data/3cfF4iQQRq60bIYSf3lmPg] already exists","index_uuid":"3cfF4iQQRq60bIYSf3lmPg","index":"airline-data","stack_trace":"[airline-data/3cfF4iQQRq60bIYSf3lmPg] org.elasticsearch.ResourceAlreadyExistsException: index [airline-data/3cfF4iQQRq60bIYSf3lmPg] already exists\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:177)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1371)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:345)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:300)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1645)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:356)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1642)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:356)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"}],"type":"resource_already_exists_exception","reason":"index [airline-data/3cfF4iQQRq60bIYSf3lmPg] already exists","index_uuid":"3cfF4iQQRq60bIYSf3lmPg","index":"airline-data","stack_trace":"[airline-data/3cfF4iQQRq60bIYSf3lmPg] org.elasticsearch.ResourceAlreadyExistsException: index [airline-data/3cfF4iQQRq60bIYSf3lmPg] already exists\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:177)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1371)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:345)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:300)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1645)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:356)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$Ba…	
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:610)	
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:544)
...
@thecoop thecoop added >test-failure Triaged test failures from CI :Analytics/ES|QL AKA ESQL v8.14.0 labels May 23, 2024
@elasticsearchmachine elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) needs:risk Requires assignment of a risk label (low, medium, blocker) labels May 23, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@thecoop thecoop changed the title [CI] [CI] Failure in org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=transform/transforms_start_stop/Test stop missing transform} May 23, 2024
@thecoop thecoop added :ml/Transform Transform and removed :Analytics/ES|QL AKA ESQL labels May 23, 2024
@elasticsearchmachine elasticsearchmachine added the Team:ML Meta label for the ML team label May 23, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@elasticsearchmachine elasticsearchmachine removed the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label May 23, 2024
@prwhelan
Copy link
Member

Failure seems to be #107505

Or at least that's how I've been debugging test failures? The first test failure is the one to fix, and the following are just a cascade of failures as the nodes shut down?

REPRODUCE WITH: ./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline}" -Dtests.seed=858DD2DE27314F95 -Dtests.locale=zh-Hant-HK -Dtests.timezone=Africa/Douala -Druntime.java=22	
org.elasticsearch.xpack.test.rest.XPackRestIT > test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline} FAILED	
    java.lang.AssertionError: Failure at [ml/inference_crud:680]: expected [2xx] status code but api [ml.delete_trained_model] returned [503 Service Unavailable] [{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]\n"}],"type":"search_phase_execution_exception","reason":"Partial shards failure","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-stats-000001","node":"NtH1xIEnSwu9mQusQ_a5UQ","reason":{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]\n"}}],"stack_trace":"Failed to execute phase [query], Partial shards failure; shardFailures {[NtH1xIEnSwu9mQusQ_a5UQ][.ml-stats-000001][0]: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]\n}\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:418)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1752)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1476)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1610)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1585)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:44)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:151)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.NoShardAvailableActionException.forOnShardFailureWrapper(NoShardAvailableActionException.java:28)\n\tat org.elasticsearch.s…	
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:610)	
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:553)	
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)	
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)	
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)	
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)	
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)	
        at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)	
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)	
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)	
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)	
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)	
        at java.base/java.lang.Thread.run(Thread.java:1570)	
        Caused by:	
        java.lang.AssertionError: expected [2xx] status code but api [ml.delete_trained_model] returned [503 Service Unavailable] [{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]\n"}],"type":"search_phase_execution_exception","reason":"Partial shards failure","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-stats-000001","node":"NtH1xIEnSwu9mQusQ_a5UQ","reason":{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]\n"}}],"stack_trace":"Failed to execute phase [query], Partial shards failure; shardFailures {[NtH1xIEnSwu9mQusQ_a5UQ][.ml-stats-000001][0]: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]\n}\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:418)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:497)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1752)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1476)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1610)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1585)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:44)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:151)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasticsearch.action.NoShardAvailableActionException.forOnShardFailureWrapper(NoShardAvailableActionException.java:28)\n\tat org.elasticsearch.server@8.14.0-SNAPSHOT/org.elasti…	
            at org.junit.Assert.fail(Assert.java:89)	
            at org.elasticsearch.test.rest.yaml.section.DoSection.checkResponseException(DoSection.java:540)	
            at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:396)	
            at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:590)	
            ... 41 more	
    org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:39213], URI [/_ml/trained_models/_stats?size=10000], status line [HTTP/1.1 500 Internal Server Error]	
    {"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]"}],"type":"exception","reason":"Searching for stats for models [h-classification-model,d-classification-model,c-classification-model,g-classification-model,i-classification-model,b-classification-model,a-classification-model,lang_ident_model_1,y-classification-model,f-classification-model,k-classification-model,j-classification-model,z-classification-model,a-regression-model-1,a-regression-model-0,e-classification-model] failed","caused_by":{"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-stats-000001","node":"NtH1xIEnSwu9mQusQ_a5UQ","reason":{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:32997][indices:data/read/search[phase/query]]"}}]}},"status":500}	
        at app//org.elasticsearch.client.RestClient.convertResponse(RestClient.java:351)	
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:317)	
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)	
        at app//org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.deleteAllTrainedModelIngestPipelines(MlRestTestStateCleaner.java:42)	
        at app//org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.resetFeatures(MlRestTestStateCleaner.java:33)	
        at app//org.elasticsearch.xpack.test.rest.AbstractXPackRestTest.clearMlState(AbstractXPackRestTest.java:138)	
        at app//org.elasticsearch.xpack.test.rest.AbstractXPackRestTest.cleanup(AbstractXPackRestTest.java:118)	
        at java.base@22.0.1/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)	
        at java.base@22.0.1/java.lang.reflect.Method.invoke(Method.java:580)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)	
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)	
        at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
        at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)	
        at app//org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)	
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)	
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)	
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
        at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
        at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)	
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)	
        at java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)	
    org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:39213], URI [/_features/_reset], status line [HTTP/1.1 500 Internal Server Error]	
    {"error":{"root_cause":[{"type":"runtime_exception","reason":"Unable to reset machine learning feature as there are ingest pipelines still referencing trained machine learning models"}],"type":"runtime_exception","reason":"Unable to reset machine learning feature as there are ingest pipelines still referencing trained machine learning models"},"status":500}	
        at app//org.elasticsearch.client.RestClient.convertResponse(RestClient.java:351)	
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:317)	
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)	
        at app//org.elasticsearch.test.rest.ESRestTestCase.wipeCluster(ESRestTestCase.java:835)	
        at app//org.elasticsearch.test.rest.ESRestTestCase.cleanUpCluster(ESRestTestCase.java:531)	
        at java.base@22.0.1/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)	
        at java.base@22.0.1/java.lang.reflect.Method.invoke(Method.java:580)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)	
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)	
        at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
        at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)	
        at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)	
        at app//org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)	
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)	
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)	
        at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
        at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
        at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)	
        at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
        at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)	
        at java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)

@nik9000
Copy link
Member

nik9000 commented May 24, 2024 via email

@prwhelan
Copy link
Member

Ah thank you, I wasn't sure if that was correct or not. When we fix the duplicate ticket, I'll double back and make sure this issue isn't seen anymore either.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:ml/Transform Transform needs:risk Requires assignment of a risk label (low, medium, blocker) Team:ML Meta label for the ML team >test-failure Triaged test failures from CI v8.14.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nik9000 @prwhelan @thecoop @elasticsearchmachine