Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] XPackRestIT test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline} failing #106652

Open
ldematte opened this issue Mar 22, 2024 · 6 comments · May be fixed by #108202
Open

[CI] XPackRestIT test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline} failing #106652

ldematte opened this issue Mar 22, 2024 · 6 comments · May be fixed by #108202
Labels
:ml Machine learning needs:risk Requires assignment of a risk label (low, medium, blocker) Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@ldematte
Copy link
Contributor

Many other tests in the same build scan (https://gradle-enterprise.elastic.co/s/uprnbtvfhrrvk) fail with the same message: "Unable to reset machine learning feature as there are ingest pipelines still referencing trained machine learning models"

Possibly related to: #71072

Build scan:
https://gradle-enterprise.elastic.co/s/uprnbtvfhrrvk/tests/:x-pack:plugin:yamlRestTest/org.elasticsearch.xpack.test.rest.XPackRestIT/test%20%7Bp0=ml%2Finference_crud%2FTest%20force%20delete%20given%20model%20with%20alias%20referenced%20by%20pipeline%7D

Reproduction line:

./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline}" -Dtests.seed=5314F766DCACC2B2 -Dtests.locale=uk -Dtests.timezone=Asia/Thimphu -Druntime.java=22

Applicable branches:
8.13

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.xpack.test.rest.XPackRestIT#test {p0=ml/inference_crud/Test force delete given model with alias referenced by pipeline}

Failure excerpt:

java.lang.AssertionError: Failure at [ml/inference_crud:680]: expected [2xx] status code but api [ml.delete_trained_model] returned [503 Service Unavailable] [{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n"}],"type":"search_phase_execution_exception","reason":"Partial shards failure","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-stats-000001","node":"yCCTleljQ7eVm9bxerfJ4Q","reason":{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n"}}],"stack_trace":"Failed to execute phase [query], Partial shards failure; shardFailures {[yCCTleljQ7eVm9bxerfJ4Q][.ml-stats-000001][0]: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n}\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:418)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.successfulShardExecution(AbstractSearchAsyncAction.java:616)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResultConsumed(AbstractSearchAsyncAction.java:600)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$onShardResult$12(AbstractSearchAsyncAction.java:583)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.QueryPhaseResultConsumer$PendingMerges.consume(QueryPhaseResultConsumer.java:383)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.QueryPhaseResultConsumer.consumeResult(QueryPhaseResultConsumer.java:116)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResult(AbstractSearchAsyncAction.java:583)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.onShardResult(SearchQueryThenFetchAsyncAction.java:120)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:324)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:33)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:18)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:62)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:26)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:48)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:628)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleResponse(TransportService.java:1742)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1465)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1562)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1536)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:35)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:32)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:19)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:289)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:73)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:70)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.NoShardAvailableActionException.forOnShardFailureWrapper(NoShardAvailableActionException.java:28)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:532)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:479)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1751)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1475)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1609)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1584)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:44)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:146)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\t... 5 more\n"},"status":503}]

  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:561)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:504)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1570)

  Caused by: java.lang.AssertionError: expected [2xx] status code but api [ml.delete_trained_model] returned [503 Service Unavailable] [{"error":{"root_cause":[{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n"}],"type":"search_phase_execution_exception","reason":"Partial shards failure","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".ml-stats-000001","node":"yCCTleljQ7eVm9bxerfJ4Q","reason":{"type":"no_shard_available_action_exception","reason":"[yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]","stack_trace":"org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n"}}],"stack_trace":"Failed to execute phase [query], Partial shards failure; shardFailures {[yCCTleljQ7eVm9bxerfJ4Q][.ml-stats-000001][0]: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n}\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:418)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:744)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.successfulShardExecution(AbstractSearchAsyncAction.java:616)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResultConsumed(AbstractSearchAsyncAction.java:600)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$onShardResult$12(AbstractSearchAsyncAction.java:583)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.QueryPhaseResultConsumer$PendingMerges.consume(QueryPhaseResultConsumer.java:383)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.QueryPhaseResultConsumer.consumeResult(QueryPhaseResultConsumer.java:116)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardResult(AbstractSearchAsyncAction.java:583)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.onShardResult(SearchQueryThenFetchAsyncAction.java:120)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:324)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:33)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:18)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:62)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:26)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:48)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:628)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleResponse(TransportService.java:1742)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1465)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1562)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1536)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:35)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:32)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:19)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListener.respondAndRelease(ActionListener.java:289)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:73)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:70)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: org.elasticsearch.action.NoShardAvailableActionException: [yamlRestTest-0][127.0.0.1:41851][indices:data/read/search[phase/query]]\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.NoShardAvailableActionException.forOnShardFailureWrapper(NoShardAvailableActionException.java:28)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:532)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:479)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:335)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:31)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:53)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:634)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1751)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1475)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1609)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1584)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:44)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:44)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:146)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.server@8.13.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\t... 5 more\n"},"status":503}]

    at org.junit.Assert.fail(Assert.java:89)
    at org.elasticsearch.test.rest.yaml.section.DoSection.checkResponseException(DoSection.java:529)
    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:385)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:541)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:504)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke(Method.java:580)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1570)
@ldematte ldematte added :ml Machine learning >test-failure Triaged test failures from CI labels Mar 22, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@elasticsearchmachine elasticsearchmachine added blocker Team:ML Meta label for the ML team labels Mar 22, 2024
@ldematte
Copy link
Contributor Author

Another set of failures: https://gradle-enterprise.elastic.co/s/6ksrvsmntrdr6

@alex-spies
Copy link
Contributor

And another one: https://gradle-enterprise.elastic.co/s/td2aksmy5mjya

@alex-spies alex-spies mentioned this issue Apr 23, 2024
Open
@elasticsearchmachine elasticsearchmachine added the needs:risk Requires assignment of a risk label (low, medium, blocker) label Apr 24, 2024
@maxhniebergall
Copy link
Member

It seems like this issue is still blocked by the underlying issue: #65846

@davidkyle davidkyle linked a pull request May 2, 2024 that will close this issue
Open
@ywangd
Copy link
Member

ywangd commented May 7, 2024

When the test fail, it seems to impact the rest of the tests running in the same suite

@nielsbauman
Copy link
Contributor

Another failing build: https://gradle-enterprise.elastic.co/s/w4esogkgaw6xk

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:ml Machine learning needs:risk Requires assignment of a risk label (low, medium, blocker) Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] Wait for all shards to be active when creating the ML stats index davidkyle/elasticsearch
7 participants
@ldematte @ywangd @mark-vieira @nielsbauman @elasticsearchmachine @alex-spies @maxhniebergall